Finding a Model
Introduction
Hundreds of language models exist today. To run one locally, you need a model that fits your hardware and performs well on your task. This article shows where to find models, which criteria matter most, and how to tell if a model is right for you.
Finding a Model - In A Nutshell
A good fit is small enough for your hardware and capable enough for your task. For getting started, models like Llama 3, Qwen 2, or Mistral in 7B or 8B parameters work well. Quantization makes them smaller without losing much quality. The main sources are the Ollama library, Hugging Face, and TheBloke model variants.
Key Terms and Components
| Term | Meaning |
|---|---|
| Parameters | Model size, typically measured in billions |
| GGUF | Container format for quantized models |
| Quantization | Reducing model precision to save storage |
| Ollama Library | Official collection of Ollama-compatible models |
| Hugging Face | Platform for AI models, datasets, and tools |
Practical Relevance
Why does selection matter?
A model that’s too large won’t run on your hardware. One that’s too small may give poor answers. A coding-focused model writes better code, while a text model handles languages like German more fluently. Model choice is the most important decision after installation.
What drives the choice?
Key factors include model size, quantization level, training data, capabilities, and license. For beginners, the first two matter most. More hardware means you can run larger or higher-quality quantized models.
Popular Starter Models
| Model | Size | Use Case | Highlight |
|---|---|---|---|
| Llama 3 | 8B, 70B | Text, code, chat | Strong general-purpose performance |
| Qwen 2 | 7B, 72B | Code, logic, multilingual | Excellent coding ability |
| Mistral | 7B, 8x7B | Text, reasoning | Efficient and fast |
| Phi 3 | 4B, 14B | Entry-level, resource-constrained | Small models with solid performance |
| DeepSeek | 7B, 67B | Coding, reasoning | Open, high-performance variants |
Where to Find Models
Three main sources exist for local models:
Ollama Library
The simplest option. Visit ollama.com/library to see available models and start one with ollama run modellname.
Hugging Face The largest model platform. You’ll find original models, quantized variants, and specialized versions. For local use, search for GGUF files.
TheBloke A well-known source for pre-built GGUF quantizations of popular models. Especially useful when Ollama doesn’t yet offer a model directly.
Decision Checklist
Ask yourself three questions before downloading:
- What do you need to do? Chat, write code, analyze documents, or understand images?
- How much storage do you have? At least 8 GB VRAM opens up many 7B models.
- What language do you need? Not all models are equally well-trained for languages like German.
If you’re unsure, start with Llama 3 8B in Q4_K_M. It’s a solid all-rounder for text and code.
Hardware, Cost, and Security Considerations
Most open-source models are free. Licenses vary, especially for commercial use. Check the license terms carefully. A major security advantage of local models is that your data stays on your machine instead of being sent to the model provider. Still, with sensitive data, verify the model runs locally and no telemetry is enabled.
More on AI and Related Topics
- Start with Ollama Library for quick setup.
- For specialized needs, use Hugging Face and GGUF files.
- Q4_K_M strikes a good balance between storage and quality.
- Model size is the biggest lever for both performance and capability.
Learn more about quantization in Quantization and hardware requirements in RAM and VRAM Requirements.
FAQ - Common Questions About Finding a Model
Is a bigger model always better?
Not necessarily. Larger models often perform better but are slower and use more memory. For many tasks, a well-quantized 7B or 8B model works fine.
What’s a good first model?
Llama 3 8B, Qwen 2 7B, or Mistral 7B are strong general-purpose options that run on most consumer systems.
Can I use models from Hugging Face in Ollama?
Yes, if they’re in GGUF format. Ollama can create modelfiles that point to GGUF files.
Are all models free?
Most open-source models are free to use. For commercial purposes, check the specific license.
Where do I find models for specific languages?
Hugging Face and Ollama Library let you filter by language or region. Multilingual models like Qwen or Llama 3 are usually good choices.
Tools and Further Reading
Beyond Ollama and Hugging Face, model comparisons, leaderboards, and benchmarks help. A good starting point is the Ollama library, since it offers ready-to-run models immediately.
Sources
- Ollama Model Library
- Hugging Face Model Catalog
- TheBloke GGUF Models