Using GPUs in Docker

What this article covers

• Why GPUs matter in Docker.
• Installing the NVIDIA Container Toolkit.
• ROCm and AMD GPUs.
• Docker run and Compose configuration.
• Troubleshooting GPU detection.

Introduction: Using GPUs in Docker

Many AI and compute-heavy applications require a GPU. Docker isolates workloads, but modern container technologies let you pass GPUs through to containers. NVIDIA provides the NVIDIA Container Toolkit for this purpose; AMD relies on ROCm. When you properly integrate GPUs into Docker, you can run containerized applications with full graphics acceleration.

This article explains how to make NVIDIA and AMD GPUs available in Docker.

Key concepts

• NVIDIA Container Toolkit: Driver and runtime for Docker.
• ROCm: AMD’s open-source platform for GPU computing.
• runtime: Container runtime that exposes the GPU.
• driver: Host-level driver.
• CUDA: NVIDIA’s compute platform.
• HIP: AMD’s equivalent to CUDA.
• /dev/nvidia0: NVIDIA device file.
• /dev/dri: AMD/Intel device files.
• capabilities: GPU device capabilities.

Preparing NVIDIA

1. Install the NVIDIA driver.
2. Install the NVIDIA Container Toolkit:

distribution=$(. /etc/os-release;echo $ID$VERSION_ID)
curl -s -L https://nvidia.github.io/nvidia-docker/gpgkey | sudo apt-key add -
curl -s -L https://nvidia.github.io/nvidia-docker/$distribution/nvidia-docker.list | \
  sudo tee /etc/apt/sources.list.d/nvidia-docker.list
sudo apt-get update
sudo apt-get install -y nvidia-container-toolkit
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

Testing NVIDIA

docker run --rm --gpus all nvidia/cuda:12-base nvidia-smi

Docker run with GPU

docker run --rm --gpus all \
  -v ollama:/root/.ollama \
  -p 11434:11434 \
  ollama/ollama

Docker Compose for NVIDIA

services:
  ollama:
    image: ollama/ollama
    ports:
      - "11434:11434"
    volumes:
      - ollama:/root/.ollama
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 1
              capabilities: [gpu]

volumes:
  ollama:

Preparing AMD/ROCm

1. Install the AMDGPU driver.
2. Install ROCm packages.
3. Launch containers with /dev/dri and /dev/kfd access.

Testing AMD

docker run --rm --device /dev/kfd --device /dev/dri \
  ollama/ollama:rocm ollama --version

Docker Compose for AMD

services:
  ollama:
    image: ollama/ollama:rocm
    devices:
      - /dev/kfd
      - /dev/dri
    volumes:
      - ollama:/root/.ollama
    ports:
      - "11434:11434"

volumes:
  ollama:

GPU support matrix

NVIDIA

• Maxwell and newer.
• Volta, Turing, Ampere, Ada Lovelace.
• Full CUDA support.

AMD

• RDNA and RDNA2 with ROCm.
• Some older Radeon models.
• Not all consumer cards officially supported.

Intel

• Arc and select integrated GPUs.
• Support is growing but less widely available.

Checking GPU inside the container

docker exec ollama nvidia-smi

Or for AMD:

docker exec ollama rocminfo

Troubleshooting

• nvidia-smi not found: Toolkit not installed.
• Docker doesn’t detect GPU: Runtime not configured.
• Driver too old: Update NVIDIA driver.
• ROCm errors: Verify you’re using the correct ROCm version.
• No /dev/dri: Check driver installation and permissions.
• Out of memory: Watch VRAM limits on the container.

Tips

• Keep your driver and toolkit up to date.
• Use base images with matching CUDA/ROCm versions.
• Test nvidia-smi on the host first.
• Look for specialized images for Intel GPUs.
• Adjust Compose syntax based on your Docker version.

Further reading

• BotServ.de Ollama in Docker
• BotServ.de Docker GPU passthrough
• BotServ.de Docker container runtime
• BotServ.de Buying GPUs for local AI

FAQ: GPU in Docker

Do I need special drivers inside the container? No, the host driver is used. The toolkit handles the passthrough.

Does GPU work in Docker on Windows? Only with WSL2 and NVIDIA GPUs; it’s not native.

How do I check for GPU inside the container? Use nvidia-smi or rocminfo.

Why doesn’t Docker detect my GPU? Most often the NVIDIA Container Toolkit or Docker runtime configuration is missing.

Can I assign multiple GPUs? Yes, with --gpus '"device=0,1"' or count: all.

Sources and references

• NVIDIA Container Toolkit: https://github.com/NVIDIA/nvidia-container-toolkit
• ROCm: https://rocm.docs.amd.com/
• Docker GPU support: https://docs.docker.com/compose/gpu-support/

Summary: Using GPUs in Docker

GPUs can be passed through to Docker containers easily using the NVIDIA Container Toolkit or ROCm. NVIDIA provides the most mature support, and AMD’s ROCm works well with compatible hardware. What matters most is keeping your host drivers current, configuring the runtime correctly, and setting up Compose properly. Once you get nvidia-smi or rocminfo running inside your container, you can harness GPU-accelerated applications like Ollama in Docker.