AnythingLLM with Ollama via Docker Compose

What this article covers

• How to start AnythingLLM and Ollama with Docker Compose.
• What a proper compose file looks like.
• Which volumes and environment variables matter.
• How to access the interface and connect your first model.

Introduction: AnythingLLM with Ollama via Docker Compose

AnythingLLM is a user-friendly interface for chatting with your own documents. Ollama provides the local language model. Together, they form a complete RAG system without requiring you to write any code. Docker Compose lets you manage both services in a single file and start them with one command.

This guide walks you through a minimal compose setup that runs in minutes. It’s ideal if you want to quickly test whether AnythingLLM fits your PDFs and documents.

Why do I need Docker Compose?

Without Docker Compose, you must start each container individually and configure their connections manually. Compose bundles both services together and ensures they can communicate. You save time and reduce errors. Stopping, updating, and backing up later becomes simpler too.

Prerequisites

• Docker and Docker Compose are installed.
• At least 8 GB of RAM; 16 GB is better.
• A machine running Linux, Windows with WSL2, or macOS.
• A folder where your compose file and volumes will live.

Compose file for AnythingLLM and Ollama

Create a file named docker-compose.yml:

services:
  ollama:
    image: ollama/ollama:latest
    container_name: ollama
    volumes:
      - ollama_data:/root/.ollama
    ports:
      - "11434:11434"
    restart: unless-stopped

  anythingllm:
    image: mintplexlabs/anythingllm:latest
    container_name: anythingllm
    ports:
      - "3001:3001"
    environment:
      - OLLAMA_BASE_URL=http://ollama:11434
    volumes:
      - anythingllm_data:/app/server/storage
    depends_on:
      - ollama
    restart: unless-stopped

volumes:
  ollama_data:
  anythingllm_data:

Save the file and run this command in the same directory:

docker compose up -d

The command downloads the images, creates the volumes, and starts both containers. In moments, AnythingLLM is available at http://localhost:3001.

Key Docker Compose concepts

• Service: A container defined in the compose file.
• Volume: Persistent storage that survives container restarts.
• Port mapping: Connects container ports to host ports.
• depends_on: Ensures Ollama starts before AnythingLLM.
• OLLAMA_BASE_URL: Tells AnythingLLM where to find Ollama.

Loading your first model

After startup, pull a model into Ollama:

docker exec -it ollama ollama pull llama3.1

Then navigate to http://localhost:3001 and select Ollama as the provider in AnythingLLM. The model appears automatically.

Who should use this guide?

• Beginners who want to quickly test a local RAG system.
• Teams setting up shared knowledge management.
• Anyone who prefers Docker and wants to avoid installing software.

Common pitfalls

• Not enough RAM: Large models consume significant memory. Smaller models like llama3.2 or phi4-mini work well for initial tests.
• Port 3001 in use: Change the port in the ports section if needed.
• Ollama unreachable: Check that the container is running and OLLAMA_BASE_URL is correct.
• Data loss: Without volumes, all documents and models disappear on restart.

Further links and resources

• AnythingLLM on BotServ.de
• Ollama on BotServ.de
• Docker Basics
• AnythingLLM Documentation

FAQ: AnythingLLM with Ollama via Docker Compose

Do I need a GPU? No, but it’s recommended. Small models run on CPU, just slower.

Can I use multiple models? Yes. Pull multiple models with ollama pull and select them in AnythingLLM.

Where do my documents live? In the Docker volumes ollama_data and anythingllm_data.

How do I stop everything? Run docker compose down. Use docker compose down -v to also remove the volumes.

Does this work on Windows? Yes, with WSL2 and Docker Desktop.

Sources and further reading

• Ollama Docker image: https://hub.docker.com/r/ollama/ollama
• AnythingLLM Docker image: https://hub.docker.com/r/mintplexlabs/anythingllm
• Docker Compose: https://docs.docker.com/compose/