Docker Fundamentals: Containers for AI Services

What This Article Covers

• What Docker is and why it’s practical for AI services
• How containers, images, volumes, and docker-compose relate to each other
• How to install Docker on Linux, macOS, and Windows
• How to start your first container and persist data
• Common pitfalls for beginners and how to avoid them

Introduction: Docker Explained Simply

Docker fundamentally changed how software is installed and run. Instead of manually setting up each program with all its dependencies, you package it in a container that contains everything it needs. This is especially valuable for AI services, which often have complex dependencies, require specific Python versions, and don’t always install cleanly.

If you want to run local AI, Docker is nearly essential. Whether it’s Ollama, Open WebUI, or a vector database like Chroma, almost every AI service offers a ready-made Docker image. You download it, start the container, and the service runs without you needing to worry about libraries, paths, or conflicts.

This article walks you through the Docker fundamentals you need for AI services. You’ll learn the key concepts, install Docker, start your first container, and understand how volumes and docker-compose work. By the end, you’ll build Ollama as a practical example using docker-compose.

For more on self-hosting, see the self-hosting overview.

Why Do I Need Docker?

Imagine you want to install Ollama on your server. Without Docker, you’d have to go through several steps. You install the right C libraries, ensure the Python version matches, set up a system service, configure paths for models, and hope nothing conflicts with other programs. When you later want to update or remove Ollama, leftover files often remain that you need to clean up manually.

With Docker, a single command does it all:

docker run -d -p 11434:11434 -v ollama-data:/root/.ollama ollama/ollama

That’s it. Docker downloads the image, starts the container, and Ollama runs isolated from the rest of your system. Want to update it? Pull the latest image and restart. Want to remove it? Stop and delete the container. No leftovers, no conflicts, no manual cleanup.

Docker solves a core problem: it works on my machine but not on yours. A container includes exactly the libraries and settings the program needs, independent of the host system. If a container runs on your server, it also runs on another server, on a NAS, or in the cloud.

Docker in Brief

Docker works like container ships. Before standardized containers existed, cargo had to be loaded individually. Sacks, barrels, crates, each item required custom handling. Standardized containers changed that. Every container has the same dimensions, fits on every ship, and loads with the same crane. Whatever cargo is inside doesn’t matter; the container ensures compatibility.

Docker does the same for software. Instead of setting up each program individually on a system, you package it in a standardized container. The container holds the program, its libraries, its configuration, and its dependencies. Docker ensures the container runs on any system that has Docker installed. The program inside doesn’t know whether it’s running on Linux, macOS, or Windows; it only sees its own isolated environment.

Who Is This Article For?

This article is for beginners who are new to Docker or only vaguely familiar with it. You don’t need to be a Linux expert, but basic terminal skills help. If you’ve run a command in the terminal and edited a file with a text editor before, that’s enough.

You’re in the right place if you want to self-host AI services like Ollama or Open WebUI and wonder what Docker is all about. You’re also here if you’ve tried Docker before but haven’t quite grasped volumes, docker-compose, and port mapping.

If you already know Docker and are looking for GPU support specifically, jump directly to GPU Support in Docker.

Key Terms

Term	Meaning
Docker	Platform that runs applications in isolated containers
Container	Running instance of an image, isolated from the host system
Image	Template for containers, contains all files and settings
Volume	Persistent storage that saves data beyond the container’s lifetime
docker-compose	Tool to define and start multiple containers via a YAML file
Port Mapping	Forward a host port to the container so services are accessible from outside
Dockerfile	Text file with instructions for building your own image
Registry	Repository for images, such as Docker Hub
Layer	Layer of an image; each instruction in the Dockerfile creates one
Network	Virtual network through which containers communicate

Installing Docker

Docker runs natively on Linux. On macOS and Windows, you use Docker Desktop, which starts a virtual machine in the background.

Linux

On Linux, install Docker best via the official installation script. On Ubuntu and other Debian-based distributions, it works like this:

curl -fsSL https://get.docker.com | sudo sh

Next, add your user to the Docker group so you can run Docker commands without sudo:

sudo usermod -aG docker $USER

Log out and back in for the change to take effect. Verify the installation with:

docker --version

macOS

On macOS, download Docker Desktop from the official website and install it like any normal application. Start Docker Desktop and wait until the whale icon in the menu bar is stable. After that, all docker commands work in the terminal.

On Macs with Apple Silicon, Docker Desktop automatically uses the ARM architecture. Make sure you pull images available for ARM. Many official images now offer linux/arm64 variants.

Windows

On Windows, also install Docker Desktop. The requirement is WSL2, the Windows Subsystem for Linux. Docker Desktop sets up WSL2 automatically if it’s not already installed. After installation, all Docker commands work in PowerShell or CMD.

GPU support on Windows works with WSL2 and an Nvidia driver that supports WSL2. More on that in GPU Support in Docker.

Getting Started

After installation, test whether Docker runs correctly. The classic first command is:

docker run hello-world

Docker downloads a small test image and runs it. If you see a welcome message, Docker works. If you get a permission error, you probably haven’t added your user to the Docker group yet or haven’t logged back in.

List all running containers:

docker ps

With -a, you also see stopped containers:

docker ps -a

Show all images you’ve downloaded:

docker images

These three commands, docker run, docker ps, and docker images, cover the main operations. You start containers, check their status, and see which images are available locally.

Volumes and Data

Containers are ephemeral. Delete a container, and all its data vanishes. For AI services, this is a problem because models are large, and you don’t want to download them again every time you restart.

Volumes solve this. A volume is persistent storage that exists independently of any container. You mount it into a container, and any data the container writes there persists even after you delete and recreate the container.

To mount a volume into a container:

docker run -d --name ollama -p 11434:11434 -v ollama-data:/root/.ollama ollama/ollama

The -v ollama-data:/root/.ollama parameter creates a named volume called ollama-data and mounts it at /root/.ollama inside the container. Ollama stores its models there. If you delete and restart the container, the models remain.

List all volumes:

docker volume ls

Remove a volume you no longer need:

docker volume rm ollama-data

As an alternative to named volumes, you can use bind mounts to connect a specific directory on your host system into the container:

docker run -d --name ollama -p 11434:11434 -v /home/user/ollama:/root/.ollama ollama/ollama

Bind mounts give you direct access to files on the host, which is useful for backups and inspection. Named volumes are managed by Docker and are the simpler choice for most use cases.

docker-compose

When you’re running just one container, docker run is enough. Once you combine multiple services like Ollama and Open WebUI, the command line becomes long and unwieldy. Enter docker-compose.

docker-compose is a tool for describing containers in a YAML file. Instead of long commands, you write a docker-compose.yml and start everything with a single command.

A simple example:

services:
  ollama:
    image: ollama/ollama:latest
    container_name: ollama
    ports:
      - "11434:11434"
    volumes:
      - ollama-data:/root/.ollama
    restart: unless-stopped

volumes:
  ollama-data:

Start the setup with:

docker compose up -d

Stop and remove:

docker compose down

The -d flag runs containers in the background. Without it, they run in the foreground and you see logs directly in your terminal, which helps when debugging.

docker-compose has several advantages. Your configuration lives as a file you can version, share, and rebuild on another machine. You define multiple services in one file, and docker-compose automatically networks them so they can communicate. You don’t need to memorize long docker run commands.

Example: Ollama with Docker

Ollama is an excellent starting point for AI services in Docker. It’s popular, has an officially maintained image, and works on both CPU and GPU. In this example, you’ll set up Ollama with docker-compose, including persistence and automatic restart.

Step 1: Create a directory

Create a directory for your Docker setup:

mkdir ~/ollama-docker
cd ~/ollama-docker

Step 2: Create docker-compose.yml

Create a file named docker-compose.yml with the following content:

services:
  ollama:
    image: ollama/ollama:latest
    container_name: ollama
    ports:
      - "11434:11434"
    volumes:
      - ollama-data:/root/.ollama
    restart: unless-stopped

volumes:
  ollama-data:

Step 3: Start the container

Start the container:

docker compose up -d

Docker downloads the image if it’s not already local and starts the container. Check the status:

docker ps

You should see a container named ollama running.

Step 4: Load a model

Pull a model into the running container:

docker exec -it ollama ollama pull llama3

Then start the model:

docker exec -it ollama ollama run llama3

An interactive chat session opens. Type /bye to end it.

Step 5: Test the API

Check if the API is reachable:

curl http://localhost:11434/

If you get Ollama is running as the response, everything works. For more on the API and integration with other tools, see Ollama with Docker.

If you want to run Ollama with GPU support, add GPU configuration to your docker-compose.yml. Details are in GPU Support in Docker.

Common Pitfalls

Permission denied when running Docker commands. You haven’t added your user to the Docker group. Run sudo usermod -aG docker $USER and log in again. Docker commands will then work without sudo.

Data disappears after container restart. You didn’t mount a volume. Without -v, the container stores everything in its own filesystem, which is lost when you delete the container. Always use a volume for data you want to keep.

Port already in use. If the port you want to map is already taken by another program, the container won’t start. Check which process is using it with sudo lsof -i :11434, stop it, or pick a different port.

Container crashes immediately after starting. This usually means a configuration error. Check the logs with docker logs containername. The error message typically shows exactly what went wrong.

Image not available for your architecture. On Macs with Apple Silicon or ARM servers, an image might only exist for x86_64. Docker will try to emulate it, which is extremely slow or won’t work at all. Check if the image offers a linux/arm64 variant.

docker-compose command doesn’t work. Older versions use docker-compose with a hyphen, newer ones use docker compose as a subcommand. Both work, but if you have a current Docker version, use docker compose.

Disk fills up. Each image and container consumes space. Old images and stopped containers accumulate over time. Clean up with docker system prune, which removes all unused images, containers, and networks.

Hardware, Costs, and Security

Docker itself is free and open source. Docker Desktop is free for personal use; larger organizations should review the license terms. Hardware requirements depend on what you run in the containers. Docker itself needs minimal resources, but AI services like Ollama need adequate RAM and ideally a GPU.

From a security perspective, a few things matter. Containers are isolated, but not automatically secure. Don’t expose ports to the internet unprotected; use a reverse proxy with authentication instead. Pull images only from trusted sources, preferably official images or verified publishers on Docker Hub. For unofficial images, read the Dockerfile before using it, especially if it handles sensitive data.

For more on secure self-hosting setups, see Secure Operations.

Further Reading

• Self-Hosting Overview to get started with the topic
• Ubuntu as a foundation for Docker servers
• GPU Support in Docker for GPU configuration
• Ollama with Docker for a complete Ollama setup
• Open WebUI as a user interface for Ollama
• Secure Deployment for self-hosting security

FAQ: Docker Basics

What’s the difference between Docker and a virtual machine?

A virtual machine emulates a complete operating system with its own kernel. A Docker container uses the kernel of the host system and contains only the application and its dependencies. Containers are therefore lighter and start faster than virtual machines.

Do I need Linux to use Docker?

No. Docker runs natively on Linux. On macOS and Windows, you use Docker Desktop, which starts a virtual machine in the background. The Docker commands are identical across all platforms.

What’s the difference between an image and a container?

An image is a static template that contains all files and settings. A container is a running instance of an image. You can start as many containers as you want from a single image.

Is data in a container safe?

Not automatically. If you delete a container, the data inside is lost. Use volumes to persist data so it survives even after you delete and recreate the container.

What does Docker cost?

Docker Engine and docker-compose are open source and free. Docker Desktop is free for personal use and small companies; larger organizations have licensing terms.

Can I use Docker without internet?

Yes, once you’ve downloaded all necessary images. Pulling images requires internet, but starting and stopping containers works offline.

What’s the difference between docker-compose and Kubernetes?

docker-compose is designed for single hosts and defines multiple containers in a YAML file. Kubernetes is a cluster orchestrator for large setups across multiple machines. For self-hosting, docker-compose is usually sufficient.

How do I update a container?

Pull the latest image with docker pull imagename, stop the running container, remove it, and start it again. If you use a volume, your data remains intact. With docker-compose, docker compose pull followed by docker compose up -d is all you need.

Can I run multiple containers on the same port?

Not on the same host port. Each port can only be bound once. You can use different host ports and map them all to the same container port, for example -p 11435:11434 for the second container.

What is a Dockerfile?

A Dockerfile is a text file with instructions to build your own image. You define which base image to use, which files to copy, and which commands to execute at startup. If you’re using ready-made images, you don’t need your own Dockerfile.

How do I see a container’s logs?

Use docker logs containername to view the container’s output. With docker logs -f containername, the logs follow in real time, which is helpful for debugging.

Sources

• Docker official documentation
• Docker Hub
• docker-compose specification
• Ollama Docker Hub repository
• NVIDIA Container Toolkit documentation