llama.cpp: The Engine Under the Hood
What this article covers
- What llama.cpp is and why it’s the central inference engine in the local AI world
- How to install llama.cpp, either as a pre-built binary or compiled from source
- Which GPU backends (CUDA, Metal, ROCm, Vulkan) are supported and how to enable them
- How to load models, run them, and use server mode for an OpenAI-compatible API
- When llama.cpp is the right choice versus Ollama or LM Studio
Introduction: Understanding llama.cpp
If you work with local AI, you’ll encounter one name repeatedly: llama.cpp. Tools like Ollama and LM Studio rely on this C++ library as their inference engine. It’s the foundation that much of the local AI ecosystem is built on.
Georgi Gerganov originally created llama.cpp to run Meta’s LLaMA model on consumer-grade hardware. Today, the project supports dozens of model architectures and runs on nearly any device, from Raspberry Pi to high-end workstations with multiple GPUs.
This article explains what llama.cpp does, how to install and use it, and when direct use makes sense instead of relying on a graphical interface. If you’re completely new to this topic, start with What is local AI?
Why do you need llama.cpp?
Imagine running a local LLM on your machine and wanting to squeeze every ounce of performance from it. You have a specific GPU with a particular CUDA version, and Ollama’s pre-built binaries don’t use all the optimizations your hardware offers. Or you need a feature that graphical interfaces don’t expose, like specific sampling parameters or a custom backend.
That’s where llama.cpp comes in. It’s the lowest layer, giving you direct control over compilation, backend selection, and inference parameters. If you want to understand what local AI software actually does under the hood, you’ll encounter llama.cpp.
Another reason: llama.cpp is the reference implementation for new model formats and quantization methods. When a novel model architecture is released, llama.cpp is often the first project to support it. Researchers and experimenters who need access to the latest developments use llama.cpp directly.
llama.cpp explained
llama.cpp is a C++ library and a set of command-line tools for inferencing Large Language Models. At its core is a pure C/C++ implementation that doesn’t depend on external deep learning frameworks like PyTorch or TensorFlow. This makes it lightweight to compile and easy to port to a wide range of platforms.
The main components are:
- llama-cli: Runs a model and generates text in interactive mode.
- llama-server: Starts an HTTP server with an OpenAI-compatible API.
- llama-quantize: Quantizes models into various GGUF formats.
- libllama: The library that other programs can link to.
Models are loaded in GGUF format, which bundles quantization and metadata in a single file. llama.cpp supports a growing list of architectures, including LLaMA, Mistral, Qwen, Phi, Gemma, and many more.
Who is llama.cpp for?
llama.cpp serves several audiences:
- Developers who want to integrate an inference engine into their own applications.
- Power users who need maximum control over parameters and performance.
- Researchers experimenting with new model architectures or quantization methods.
- System administrators operating a local inference server with an OpenAI-compatible API.
If you just want to quickly test a model without wrestling with build systems and command-line flags, Ollama or LM Studio are better choices. These tools use llama.cpp under the hood but wrap it in a user-friendly interface. See Running LLMs locally for more details.
Key terms around llama.cpp
| Term | Explanation |
|---|---|
| llama.cpp | C++ inference engine for LLMs, independent of PyTorch or TensorFlow |
| GGUF | File format for quantized models, supersedes the older GGML format |
| GGML | The original model format, now replaced by GGUF |
| Quantization | Reducing the precision of model weights to save memory and compute time |
| CUDA | Nvidia’s platform for GPU programming, the primary GPU backend for llama.cpp |
| Metal | Apple’s GPU API, used for Macs with Apple Silicon or AMD GPUs |
| BLAS | Basic Linear Algebra Subprograms, libraries for optimized matrix operations |
| AVX | Advanced Vector Extensions, CPU instruction set for parallel floating-point operations |
| KV-Cache | Intermediate storage for key and value tensors, accelerates text generation |
| Context Window | Maximum number of tokens the model can process simultaneously |
What makes llama.cpp special?
Three characteristics set llama.cpp apart from other inference engines:
CPU optimization as core competency. llama.cpp was designed from the start to run efficiently on CPUs. It uses SIMD instruction sets like AVX, AVX2, AVX-512, and NEON to accelerate the matrix operations in inference. On a modern CPU, you can run a 7B model in 4-bit quantization at acceptable speeds without a GPU. See CPU vs. GPU for details.
Quantization as a first-class feature. Instead of loading models in full 16-bit precision, llama.cpp uses quantized GGUF files. The most common formats are Q4_K_M, Q5_K_M, and Q8_0, which balance size, speed, and quality. You can quantize models yourself or download ready-made GGUF files from platforms like Hugging Face. Learn more about the theory in Quantization.
Broad hardware support. llama.cpp runs on x86, ARM, WebAssembly, and RISC-V. It supports Nvidia’s CUDA, Apple’s Metal, AMD’s ROCm, Vulkan, and OpenCL as GPU backends. This diversity makes it the Swiss Army knife of local inference.
Installation: Pre-built binaries
The quickest way to get llama.cpp is through pre-built binaries. You’ll find ready-made builds for Windows, macOS, and Linux on the GitHub repository’s Releases page.
Windows: Download llama-bXXXX-bin-win-cuda-cu12.x-x64.zip if you have an Nvidia GPU, or the CPU variant llama-bXXXX-bin-win-avx2-x64.zip. Extract the archive and run the included .exe files.
macOS: Macs with Apple Silicon have builds with Metal support. Download llama-bXXXX-bin-macos-arm64.zip. On Intel Macs, use the x64 variant.
Linux: The selection is largest here. Builds are available for CPU, CUDA, and ROCm. Choose the variant that matches your hardware.
Keep in mind that pre-built binaries aren’t optimal for every combination of CPU instruction set and GPU driver. If you want to squeeze the last bit of performance, compiling from source is the better approach.
Building from Source
Compiling llama.cpp is straightforward thanks to CMake. You’ll need a C++ compiler, CMake, and Git.
CPU-only Build
git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp
cmake -B build
cmake --build build --config ReleaseAfter compilation, you’ll find the binaries in the build/bin/ directory.
CUDA Build (Nvidia)
For CUDA, install the CUDA Toolkit first. Enable CUDA with a CMake flag:
cmake -B build -DGGML_CUDA=ON
cmake --build build --config Release -jThe -j flag uses all available CPU cores during the build, significantly speeding up the process.
Metal Build (macOS)
On Macs with Apple Silicon, Metal is enabled by default. A standard build will work:
cmake -B build
cmake --build build --config ReleaseIf Metal isn’t detected automatically, you can enable it explicitly:
cmake -B build -DGGML_METAL=ON
cmake --build build --config ReleaseVulkan Build
Vulkan is useful if you have a GPU that supports neither CUDA nor Metal. Enable it like this:
cmake -B build -DGGML_VULKAN=ON
cmake --build build --config ReleaseYou’ll need the Vulkan SDK installed on your system.
GPU Support
llama.cpp supports multiple GPU backends that you can enable depending on your hardware:
CUDA (Nvidia): The most mature and performant GPU backend. If you have an Nvidia GPU, CUDA is your first choice. You can offload parts of the model to the GPU and compute the rest on the CPU. Learn more in the article GPU-Offloading.
Metal (Apple): On Macs with M1 through M4 chips, llama.cpp uses Metal for GPU acceleration. Apple Silicon’s unified memory architecture is a major advantage here, since no explicit transfer between CPU and GPU memory is required.
ROCm (AMD): For AMD GPUs, especially on Linux. ROCm is less mature than CUDA but is actively being developed.
Vulkan: A cross-platform backend that works on a wide range of GPUs. Performance is typically lower than CUDA, but compatibility is broader.
You can compile multiple backends at once. llama.cpp will select the best available backend at runtime.
Loading and Running Models
After installation, you’ll need a model in GGUF format. Download one from Hugging Face, such as a quantized Llama or Mistral variant.
Basic usage looks like this:
./llama-cli -m model.gguf -p "Explain quantization in three sentences." -n 256Key flags:
-m: Path to the GGUF model file.-p: The prompt to process.-n: Maximum number of tokens to generate.-c: Context window size; default is 2048.-t: Number of CPU threads.-ngl: Number of layers to offload to the GPU.
For interactive conversation, use -i:
./llama-cli -m model.gguf -i -c 4096 -ngl 33Here, 33 layers are offloaded to the GPU, which is typical for a 7B model with 4-bit quantization.
Server Mode
llama.cpp includes an HTTP server that provides an OpenAI-compatible API. This is useful if you have applications already using the OpenAI API and want to switch to a local model.
Start the server like this:
./llama-server -m model.gguf -c 4096 -ngl 33 --port 8080The API will then be available at http://localhost:8080. An example request:
curl http://localhost:8080/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "local-model",
"messages": [{"role": "user", "content": "Hello, who are you?"}]
}'The server supports streaming, function calling (with suitable models), and multiple parallel requests. For integrating into your own applications, server mode is often the simplest approach.
llama.cpp vs. Ollama vs. LM Studio
| Feature | llama.cpp | Ollama | LM Studio |
|---|---|---|---|
| Target Audience | Developers, power users | Beginners, developers | Beginners, visual users |
| Interface | Command line | Command line, API | GUI desktop app |
| Installation | Build or binary | Single command | Download, drag-and-drop |
| Model Management | Manual (GGUF files) | Automatic (model registry) | Automatic (Hugging Face integration) |
| GPU Support | All backends, full control | CUDA, Metal (pre-built) | CUDA, Metal (pre-built) |
| API | OpenAI-compatible (server) | OpenAI-compatible | OpenAI-compatible |
| Customization | Very high | Moderate | Low |
Use llama.cpp when: You need maximum control, want to configure specialized hardware, or are embedding llama.cpp into your own project.
Use Ollama when: You want to get started quickly and prefer a clean CLI with API support. Ollama uses llama.cpp under the hood and abstracts away the complexity.
Use LM Studio when: You prefer a graphical interface and want to download and test models with a few clicks. LM Studio also uses llama.cpp as its inference engine.
Common Pitfalls with llama.cpp
1. Wrong CPU instruction set. If you use a binary compiled for AVX2 but your CPU only supports AVX, the program will crash or run extremely slowly. Check which instruction sets your CPU supports using lscpu on Linux.
2. Insufficient GPU memory. If you offload more layers to the GPU with -ngl than available memory, inference will fail. Reduce the value incrementally until the model runs.
3. Context window too small. A context window that’s too small causes the model to forget the beginning of long prompts. Set -c to a sufficiently high value, such as 4096 or 8192, depending on the model.
4. Wrong GGUF variant. There are several quantization levels. Q4_K_M is a good default, but if you need the highest quality, use Q8_0. Q2 or Q3 are very compact but come with noticeable quality loss.
5. Outdated GGUF files. The GGUF format evolves over time. Very old GGUF files may no longer be compatible with the current llama.cpp version. Re-download models or convert them with llama-quantize.
6. Threads misconfigured. More threads aren’t always better. As a rule of thumb, use as many threads as you have physical cores, no more. Hyperthreads can even degrade performance.
7. CUDA version mismatch. The CUDA version of your binary must match your installed Nvidia driver version. If you use a CUDA 12.x binary, you need a driver that supports CUDA 12.
8. Metal not detected. On older macOS versions or with certain Xcode versions, Metal may not be found. Ensure your macOS and Xcode are up to date.
Hardware, Costs, and Security with llama.cpp
Hardware: llama.cpp runs on almost any hardware. For acceptable performance with 7B models, you need at least 8 GB of RAM and a CPU with AVX2. A GPU with 8 GB VRAM enables GPU offloading and significantly faster inference. For 13B or 33B models, 16 GB or 32 GB of RAM or VRAM respectively are recommended.
Costs: llama.cpp is open source under the MIT license and completely free. The only costs come from the hardware you run it on. There are no API fees or subscription charges, which is a major advantage over cloud-based AI services.
Security: Since llama.cpp runs locally, your prompts and data never leave your machine. This is especially relevant for sensitive data in fields like medicine, law, or business-internal information. You don’t need to send any data to external servers. Still, make sure models you download come from trusted sources, as GGUF files could theoretically be tampered with.
Further Reading and Resources on llama.cpp
- llama.cpp on GitHub - Official repository with documentation and releases
- GGUF Models on Hugging Face - Wide selection of pre-built GGUF files
- llama.cpp Discussions - Community support and tips
- Ollama - User-friendly interface built on llama.cpp
- LM Studio - Graphical desktop app powered by llama.cpp
- Quantization - Background on GGUF and quantization levels
- GPU Offloading - How to move model layers to your GPU
FAQ: llama.cpp - Common Questions
What exactly is llama.cpp? llama.cpp is a C++ library and set of command-line tools for running inference on Large Language Models. It requires no deep learning frameworks like PyTorch and compiles easily.
Do I need a GPU for llama.cpp? No. llama.cpp runs fine on CPU alone. A GPU significantly speeds up inference, but it’s not required.
What’s the difference between GGML and GGUF? GGML was llama.cpp’s original model format. GGUF is the successor, addressing several GGML limitations like missing metadata and compatibility issues. Current llama.cpp versions use GGUF.
Can I embed llama.cpp in my own application? Yes. llama.cpp is provided as a library (libllama) that you can use directly in C and C++, or via bindings in Python, Go, Rust, and other languages.
How many layers should I offload to the GPU? It depends on available VRAM. Start with a small value and increase gradually until VRAM is nearly full. For a 7B model with Q4 quantization, all 32 layers typically fit on an 8-GB GPU.
Is llama.cpp faster than Ollama? Ollama uses llama.cpp as its engine, so raw inference speed is comparable. A self-compiled llama.cpp can be slightly faster through hardware-optimized flags.
Can llama.cpp load multiple models simultaneously? Server mode can manage multiple models if you start them on different ports. True multi-model inference in a single process isn’t currently supported.
Which model architectures does llama.cpp support? A growing list including LLaMA, LLaMA 2, LLaMA 3, Mistral, Mixtral, Qwen, Phi-2, Phi-3, Gemma, Falcon, Baichuan, and many more. Check the documentation for the complete list.
How do I update llama.cpp? If you compiled from source, run git pull and recompile. For pre-built binaries, download the latest version from the releases page.
Is llama.cpp free? Yes, llama.cpp is open source under the MIT license. You can use it free of charge, including commercially.
Do I need internet to use llama.cpp? Only to download llama.cpp itself and GGUF models. Inference runs completely offline on your machine.
References and Further Reading
- Gerganov, G. (2023). llama.cpp [Software]. GitHub. https://github.com/ggerganov/llama.cpp
- GGUF Format Specification. Hugging Face. https://github.com/ggerganov/ggml/blob/master/docs/gguf.md
- llama.cpp Build Documentation. GitHub. https://github.com/ggerganov/llama.cpp/blob/master/docs/build.md
- CUDA Toolkit Documentation. Nvidia. https://docs.nvidia.com/cuda/
- Metal Performance Shaders. Apple Developer. https://developer.apple.com/metal/