Proxmox for Local AI

What this article covers

Why Proxmox works well for AI servers.
The difference between VMs and LXC containers.
How GPU passthrough enables AI workloads.
Practical backup and recovery strategies.

Introduction: Proxmox for local AI

Proxmox VE is an open-source virtualization platform. It combines KVM for virtual machines and LXC for lightweight containers. If you want to run an AI server at home or in a rack, Proxmox lets you cleanly separate multiple services: Ollama, RAG, reverse proxy, web UI, and automation each run in their own container or VM.

Why Proxmox for AI?

Proxmox provides a central web interface to manage storage, networking, resources, and backups. You can snapshot before updates, allocate resources deliberately, and quickly test new services. For AI workloads, GPU passthrough is critical: a VM gets direct access to the graphics card.

VMs vs. LXC

VMs run on KVM and are fully isolated. They’re suitable for Ollama or vLLM when you need direct GPU access.

LXC containers are lighter and consume fewer resources. They share the host kernel. They work well for web frontends, reverse proxies, monitoring, or small Python services.

For a typical local AI setup, a hybrid approach makes sense: Ollama in a VM with GPU passthrough, Open WebUI and RAG in LXC containers.

Setting up GPU passthrough

GPU passthrough requires IOMMU and VT-d enabled in the BIOS. Proxmox groups PCI devices and allows you to pass them through to a VM. With a single GPU and no host desktop, this is usually straightforward. On systems with multiple GPUs, pay attention to resource groups and boot behavior.

Ollama running inside a VM recognizes the passed-through GPU as a physical card. Performance is nearly identical to bare-metal operation.

Backup and recovery

Proxmox includes an integrated backup tool. Regular backups of VMs and containers can be automated easily. Critical data such as configurations, models, and vector databases should additionally be mirrored to separate storage, since backups otherwise consume a lot of space repeatedly.