Docker Swarm for Local AI

What This Article Covers

• What Docker Swarm is and when it makes sense to use it.
• How to initialize a Swarm cluster.
• How services, tasks, and overlay networks work.
• Practical example with Ollama or Open WebUI.
• Pros and cons compared to Kubernetes and Compose.

Introduction: Docker Swarm for Local AI

Docker Swarm is Docker’s built-in clustering feature. You can connect multiple Docker hosts into a Swarm, then distribute and scale services across them. For home labs, Swarm is often simpler than Kubernetes but more powerful than running Docker Compose on a single machine. If you have several servers and want to distribute AI services like Ollama, Open WebUI, or databases, Swarm can serve as your entry point into container orchestration.

This article shows how to set up Docker Swarm, define services, and determine when it’s worth using.

Key Concepts

• Swarm: A cluster of multiple Docker nodes.
• Manager: A control node in the Swarm.
• Worker: An execution node in the Swarm.
• Service: A defined application running in the Swarm.
• Task: A running instance of a service.
• Replica: The number of tasks running in parallel.
• Overlay network: A network spanning multiple nodes.
• Stack: A group of services defined in a Compose file.

When to Use Docker Swarm

• You have multiple physical or virtual servers.
• Services need to be highly available.
• Load should be distributed across multiple nodes.
• You want rolling updates with zero downtime.
• You need something simpler than Kubernetes to start with.

If you’re running only a single server, Docker Compose is sufficient.

Initialize the Swarm

On the first manager node:

docker swarm init --advertise-addr 192.168.1.10

The command outputs a join token. Worker and additional manager nodes join using this token:

docker swarm join --token <TOKEN> 192.168.1.10:2377

Check the status:

docker node ls

Create a Service

docker service create --name ollama \
  --publish 11434:11434 \
  --replicas 1 \
  --mount type=volume,source=ollama-data,target=/root/.ollama \
  ollama/ollama

Deploy a Stack from a Compose File

A docker-compose.yml can be deployed directly as a stack:

docker stack deploy -c docker-compose.yml ki-stack

Important: Swarm doesn’t support all Compose features. The build directive, for example, is not supported. Images must be built beforehand and stored in a registry.

Compose File for Swarm

version: "3.8"

services:
  ollama:
    image: ollama/ollama:latest
    volumes:
      - ollama-data:/root/.ollama
    networks:
      - ki-net
    deploy:
      replicas: 1
      placement:
        constraints:
          - node.labels.gpu == true

  open-webui:
    image: ghcr.io/open-webui/open-webui:main
    ports:
      - "8080:8080"
    environment:
      - OLLAMA_BASE_URL=http://ollama:11434
    networks:
      - ki-net
    deploy:
      replicas: 1

volumes:
  ollama-data:

networks:
  ki-net:
    driver: overlay

Placement and Constraints

To ensure Ollama runs on a GPU-equipped node, use labels:

docker node update --label-add gpu=true node1

In the service definition:

deploy:
  placement:
    constraints:
      - node.labels.gpu == true

Scaling

docker service scale ki-stack_open-webui=3

Or in the Compose file:

deploy:
  replicas: 3

Rolling Updates

docker service update --image ollama/ollama:latest ki-stack_ollama

Updates can be rolled out automatically in sequence across the nodes.

Secrets and Configs

Docker Swarm includes built-in secrets:

echo "mein-passwort" | docker secret create db_password -

In the service:

services:
  app:
    secrets:
      - db_password

secrets:
  db_password:
    external: true

Overlay Networks

Services on the same overlay network can reach each other by service name, regardless of which node they’re running on.

docker network create --driver overlay ki-net

Advantages Over Compose

• High availability.
• Distribution across multiple nodes.
• Rolling updates.
• Built-in load balancing.
• Overlay networks.
• Secrets and configs.

Disadvantages Compared to Kubernetes

• Smaller ecosystem.
• Limited orchestration features.
• No built-in service mesh.
• Fewer monitoring and storage integrations.

For many home labs, Swarm is sufficient. If your requirements grow more complex, you can migrate to Kubernetes later.

Common Pitfalls

• Images must be in a registry: The build directive is not executed in stacks.
• Volumes local to the node: Data does not automatically migrate with the container.
• Incorrect placement: A service might start on a node without GPU or sufficient resources.
• Firewall issues: Swarm communication requires specific ports to be open.
• Multiple managers: Use an odd number for quorum; at least three is recommended.
• No persistence strategy: Stateful services need shared storage or bind mounts.

Further Reading and Resources

• BotServ.de Docker Commands
• BotServ.de Docker Networking
• BotServ.de Docker Compose
• BotServ.de Docker Registry
• BotServ.de Kubernetes

FAQ: Docker Swarm

Do I need Swarm for a single server? No, plain Docker Compose is enough.

Is Docker Swarm outdated? Docker Swarm is still maintained, but it has less community momentum than Kubernetes.

Can I use GPU in Swarm? Yes, through placement constraints and the Nvidia Runtime on the respective nodes.

Are Swarm services highly available? Yes, when multiple replicas run on different nodes.

Which is better: Swarm or Kubernetes? Swarm is simpler, Kubernetes is more powerful. For home labs, Swarm is often sufficient.

Sources and Further Reading

• Docker Swarm Docs: https://docs.docker.com/engine/swarm/
• Docker Stack Deploy: https://docs.docker.com/engine/reference/commandline/stack_deploy/
• Docker Swarm Networking: https://docs.docker.com/network/overlay/

Summary: Docker Swarm for Local AI

Docker Swarm is a good way to distribute AI services across multiple servers without the complexity of Kubernetes. With simple commands, you can initialize a cluster, deploy services, scale them, and roll out updates. Key considerations include placement constraints for GPU workloads, overlay networks for inter-service communication, and a persistence strategy for data. For home labs with multiple nodes, Swarm provides straightforward high availability and scaling, while single-server setups often stick with Docker Compose.