Proxmox for AI: Virtualization

What This Article Covers

• How to use Proxmox VE for AI services.
• When to choose VMs versus LXC containers.
• How GPU passthrough works.
• Running Ollama in a VM or container.
• Common pitfalls with Proxmox.

Introduction

Proxmox VE is an open-source virtualization platform that excels at setting up an AI server while keeping different services cleanly isolated. You can run Ollama in one VM, a RAG service in another, and a reverse proxy in an LXC container. This keeps your setup organized and easier to maintain.

Why Proxmox for AI?

If you’re only running a single AI service, virtualization isn’t necessary. Once you have multiple workloads, Proxmox becomes valuable. You can isolate services, allocate resources precisely, and create snapshots before updates. If an experiment goes wrong, you roll back instantly. For AI workloads, GPU passthrough is the critical feature that gives a VM direct access to your graphics card.

Proxmox at a Glance

Proxmox VE combines KVM for full virtual machines and LXC for lightweight containers. Everything is managed through a web interface. Storage, networking, and resources are centrally controlled. For AI work, PCI passthrough and RAM allocation are especially important so VMs have enough power for running models.

Who This Article Is For

• Users wanting to virtualize a home server or dedicated machine.
• Proxmox beginners interested in AI.
• Anyone looking to isolate Ollama or AI tools from other services.

Key Terms

Term	Meaning
Proxmox VE	Virtualization platform combining KVM and LXC
KVM	Kernel-based Virtual Machine, full VMs
LXC	Linux Containers, lightweight containers without their own kernel
VM	Virtual machine with its own operating system
GPU Passthrough	Graphics card passed directly to a VM
IOMMU	Hardware feature for PCI device isolation
PCI	Interface for expansion cards like GPUs
Ceph	Storage solution for clusters, optional
ISO Image	Installation image for VMs
Template	Blueprint for quick VM or container creation

Installing Proxmox

1. Download the Proxmox VE ISO.
2. Create a bootable USB stick.
3. Boot the server and install Proxmox.
4. Select disk, password, and network settings.
5. After reboot, access the web interface at https://IP:8006.

Creating Your First VM for Ollama

1. Upload an Ubuntu or Debian ISO.
2. Create a new VM with at least 4 cores, 32 GB RAM, and 100 GB storage.
3. Install Linux in the VM.
4. Install Ollama:

curl -fsSL https://ollama.com/install.sh | sh
ollama run llama3.2

Setting Up GPU Passthrough

NVIDIA GPUs require IOMMU enabled in BIOS. Enable VT-d or AMD-Vi depending on your CPU. In Proxmox, modify the kernel command line:

# /etc/default/grub
GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on iommu=pt"

update-grub
reboot

Afterward, you can add the GPU to your VM as a PCI device. This isn’t straightforward and depends heavily on your motherboard and GPU model.

LXC Containers Instead of VMs

LXC containers start faster and consume less RAM. For lightweight services like a reverse proxy, Open WebUI, or small APIs, LXC containers are ideal. For Ollama with GPU support, a VM is usually better since GPU passthrough is simpler.

Networking in Proxmox

Proxmox uses Linux bridges. For getting started, simple bridged networking suffices. Assign each VM or container its own IP on your home network. This lets you apply firewall rules to individual services.

Common Pitfalls

• IOMMU not enabled: Without IOMMU in BIOS, no GPU passthrough.
• GPU bound to host: NVIDIA drivers sometimes need to be unloaded from the host before passthrough works.
• Insufficient RAM: VMs running Ollama need plenty of memory. Plan for 32 GB or more per VM.
• Disk space: Models are large. Allocate at least 100 GB per VM, preferably 500 GB.
• CPU pinning: Lock specific CPU cores to ensure the VM runs stably.
• Snapshots before updates: Always create a snapshot before testing a new model or update.
• Networking misconfiguration: Ensure VMs and containers are on the correct VLAN.
• License is free: Proxmox is open source, but support contracts are paid. Without support, you’ll see a notification in the web interface.
• USB passthrough: USB devices sometimes need manual passthrough configuration.
• Backup strategy: Proxmox Backup Server is optional but recommended.

Hardware, Costs, and Security

Proxmox doesn’t require a Windows server. It runs on consumer hardware. RAM and CPU cores are decisive. GPU passthrough requires a compatible motherboard. The software itself is free. Support contracts are optional. Security comes from network segmentation, backups, and separate VMs per service.

Further Reading

• Self-Hosting Overview
• Ubuntu for AI
• Debian for AI
• Docker Basics
• Ollama on Linux
• CPU vs GPU
• GPU Offloading

FAQ

Do I need Proxmox to run Ollama?

No, Ollama runs directly on Linux. Proxmox helps when you want to isolate multiple services.

Does Proxmox run on consumer hardware?

Yes, many people run Proxmox on home servers. Just verify IOMMU support for GPU passthrough.

Can I pass through any GPU?

Not every GPU or motherboard supports passthrough. Check IOMMU groups and motherboard support beforehand.

What’s better for Ollama: VM or LXC?

VM is better if you need GPU passthrough. LXC is lighter but works poorly with GPU passthrough.

How much RAM does an AI VM need?

7B models need 16 GB, 13B models need 32 GB, and larger models need 64 GB or more.

Is Proxmox free?

Yes, the software is open source. Support contracts are optional.

Can I use multiple GPUs?

Yes, you can assign each GPU individually to a VM as long as IOMMU is properly configured.

How do I secure my VMs?

Use snapshots, external backups, and network segmentation. Proxmox Backup Server is a good option.

Is Proxmox worth it for a single AI service?

Not really. For a single service, plain Linux directly on hardware is sufficient.

How do I access the web interface?

By default, visit https://SERVER-IP:8006. Log in with root and your installation password.

Sources

• Proxmox VE Documentation (proxmox.com)
• Ollama Linux Installation (ollama.com)
• PCI Passthrough Guide (archlinux.org)
• KVM and IOMMU Primer (kernel.org)