Ubuntu for AI: Server Setup Step by Step

What This Article Covers

Installing Ubuntu Server from ISO to first boot
Initial setup with user accounts, SSH keys, and secure root login
Updates and automated maintenance using apt and unattended-upgrades
Installing AI software like Ollama, Docker, and Python
Complete walkthrough of setting up an AI server, including common pitfalls

Introduction: Ubuntu for AI

Ubuntu is one of the most popular Linux distributions for servers and developer machines. If you want to build a local AI, Ubuntu provides a solid, well-documented foundation. You get current packages, long-term support versions, and a massive community that has already solved nearly every problem you’ll encounter.

In this article, I’ll walk you through every step needed to set up an Ubuntu Server as a platform for local AI. You’ll learn how to install the system, secure it, and equip it with AI software. By the end, you’ll have a server capable of running models like Llama or Mistral locally, without relying on cloud services or subscriptions.

This article is aimed at beginners who have never set up a Linux server before. You don’t need prior experience with Ubuntu, but you should be comfortable using the command line.

Why Ubuntu for AI?

Many Linux distributions exist, but Ubuntu offers specific advantages for AI projects. First, most AI tools like Ollama, Docker, and Python libraries publish installation instructions primarily for Ubuntu. Second, Ubuntu’s LTS versions (Long Term Support) provide five years of security updates without requiring a distribution upgrade. Third, the documentation and community are so extensive that you’ll find solutions to nearly every problem.

Other distributions like Debian or Fedora work just as well, but Ubuntu reduces friction especially for newcomers. If you later decide to switch to a different Linux variant, that’s perfectly straightforward. Ubuntu simply offers the most direct path for getting started.

If you’re interested in self-hosting more broadly, check out Self-Hosting for more articles about running your own servers.

Ubuntu in Brief

Ubuntu is a Linux distribution based on Debian and released by Canonical. There’s a desktop version with a graphical interface and a server version that runs entirely from the command line. For an AI server, you’ll use the server version because it consumes fewer resources and leaves more memory available for models.

Ubuntu releases a new version every six months. For servers, an LTS version is recommended; these release every two years and receive security updates for five years. The current relevant LTS versions are Ubuntu 24.04 LTS and Ubuntu 22.04 LTS.

Who This Article Is For

This article is intended for beginners who want to set up an Ubuntu Server for local AI. You should know how to open a terminal and run commands. Prior experience with Linux servers is helpful but not required.

If you already run an Ubuntu Server and just want to install AI software, you can skip directly to the section on AI software. If you’re starting from scratch, work through each step in order.

Key Terms

Term	Definition
Ubuntu	Debian-based Linux distribution released by Canonical
LTS	Long Term Support, a version that receives security updates for five years
SSH	Secure Shell, protocol for encrypted remote connection to a server
apt	Package manager for Ubuntu to install and update software
sudo	Command to execute another command with root privileges
systemd	Init system and service manager on modern Linux distributions
UFW	Uncomplicated Firewall, a simple firewall tool on Ubuntu
User	Regular user account with restricted permissions
Root	Administrator account with full system permissions
Kernel	Core of the operating system that bridges hardware and software

Installing Ubuntu Server

Download Ubuntu Server

Download the latest LTS version from the official Ubuntu website. You’ll receive an ISO file that you’ll write to a USB stick.

# Example: Download ISO (adjust URL as needed)
wget https://releases.ubuntu.com/24.04/ubuntu-24.04-live-server-amd64.iso

Make sure you download the server version, not the desktop version. The server version comes without a graphical interface and is designed for pure server deployment.

Create a Bootable USB Stick

Write the ISO file to a USB stick. On Linux, use dd; on Windows, you can use Rufus.

# Identify the USB stick
lsblk

# Write ISO to the USB stick (adjust device name!)
sudo dd if=ubuntu-24.04-live-server-amd64.iso of=/dev/sdX bs=4M status=progress

Replace /dev/sdX with your USB stick’s actual device name. An incorrect device name can destroy data on another drive, so double-check the output from lsblk carefully.

Perform the Installation

Boot your target machine from the USB stick and follow the installer. The key steps are:

Select language and keyboard layout
Configure networking, preferably via DHCP
Select the disk and confirm partitioning
Create a user profile: name, username, and password
Enable the SSH server so you can manage the server remotely
Complete the installation and restart

After reboot, you’ll see a login prompt. Log in with the user you created during installation.

Initial Setup

User and sudo

You already created a user during installation. This user is a member of the sudo group and can execute commands with root privileges. Verify this with:

groups

If sudo appears in the output, everything is correct. If you want to create another user:

# Create a new user
sudo adduser kiuser

# Add user to the sudo group
sudo usermod -aG sudo kiuser

Set Up SSH Keys

Instead of logging in with a password, use SSH keys. This is more secure and more convenient. Generate a key pair on your local machine:

# Run this on your local machine
ssh-keygen -t ed25519 -C "your@email.com"

Copy the public key to the server:

ssh-copy-id kiuser@server-ip

From now on, you can log in without a password, as long as your private key is available.

Once SSH keys are working, disable direct root login. Edit the SSH configuration:

sudo nano /etc/ssh/sshd_config

Change this line:

PermitRootLogin no

Restart the SSH service:

sudo systemctl restart ssh

Open a second shell and test that you can still log in before closing the first one. This prevents you from locking yourself out.

Enable the Firewall

Ubuntu includes UFW, a straightforward firewall. Enable it and allow only SSH:

sudo ufw allow ssh
sudo ufw enable

When you add services like Ollama or Docker later, open their ports explicitly. For more details, see the Firewall article.

Updates and Maintenance

apt update and upgrade

Keep your system current by applying updates regularly. The two essential commands:

# Refresh package sources
sudo apt update

# Upgrade all packages to the latest version
sudo apt upgrade

Run these commands after installation, then weekly thereafter. Security updates are particularly important because they patch known vulnerabilities.

Set Up Automatic Updates

Use unattended-upgrades for automatic security patching. This lets Ubuntu install critical updates without your intervention.

# Install the package
sudo apt install unattended-upgrades

# Enable the configuration
sudo dpkg-reconfigure -plow unattended-upgrades

Confirm the prompt with yes. From now on, the system automatically installs security updates. Regular feature updates are skipped to keep your server stable.

Install AI Software

Install Ollama

Ollama is the simplest way to run local language models on Ubuntu. Installation uses an official script:

curl -fsSL https://ollama.com/install.sh | sh

After installation, Ollama runs automatically as a systemd service. Download your first model:

ollama pull llama3.2
ollama run llama3.2

For detailed guidance, see Ollama on Linux and the Ollama overview.

Install Docker

Docker is useful if you want to run AI applications in containers. Install it from the official Docker repository:

# Add Docker repository
sudo install -m 0755 -d /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg

# Update package sources and install Docker
sudo apt update
sudo apt install docker-ce docker-ce-cli containerd.io docker-compose-plugin

Add your user to the Docker group so you can run Docker without sudo:

sudo usermod -aG docker kiuser

Log out and back in for the group change to take effect. See Docker Basics for foundational concepts.

Install Python

Many AI libraries are built on Python. Ubuntu 24.04 comes with Python 3.12. Check the version:

python3 --version

Install pip and venv for package management and virtual environments:

sudo apt install python3-pip python3-venv

Create a dedicated virtual environment for each AI project so packages don’t conflict:

python3 -m venv ~/ki-projekt
source ~/ki-projekt/bin/activate
pip install transformers torch

Example: Setting Up an AI Server

Here’s a complete walkthrough combining all steps into a working AI server.

Step 1: Update the System

sudo apt update && sudo apt upgrade -y

Step 2: Enable the Firewall

sudo ufw allow ssh
sudo ufw allow 11434  # Ollama API
sudo ufw enable

Step 3: Install Ollama

curl -fsSL https://ollama.com/install.sh | sh

Step 4: Load and Test a Model

ollama pull llama3.2
ollama run llama3.2 "Explain Ubuntu in three sentences."

Step 5: Make Ollama Accessible Over the Network

By default, Ollama listens only on localhost. If you want to access it from other machines, edit the systemd service:

sudo systemctl edit ollama.service

Add these lines:

[Service]
Environment="OLLAMA_HOST=0.0.0.0"

Restart the service:

sudo systemctl daemon-reload
sudo systemctl restart ollama

Step 6: Install Docker for Additional AI Tools

sudo apt install docker-ce docker-ce-cli containerd.io docker-compose-plugin -y
sudo usermod -aG docker kiuser

Step 7: Check Status

systemctl status ollama
systemctl status docker
ollama list

If all commands run without errors, your AI server is ready.

Common Pitfalls

Wrong USB stick with dd: Using the wrong device name overwrites another disk. Check lsblk before every write operation.
SSH login not tested after configuration changes: If you disable root login and your SSH config has errors, you may lock yourself out. Keep a second shell open until you confirm the changes work.
Firewall enabled without SSH rule: If you run ufw enable before allowing SSH, you’ll lose access. Permit SSH first, then enable the firewall.
Ollama listening only on localhost: If you want to access the API from another machine, set OLLAMA_HOST to 0.0.0.0, otherwise the service won’t be reachable from outside.
Docker without group adjustment: Without the Docker group, you’ll need to type sudo docker every time. Remember to log out and back in for the change to take effect.
Insufficient RAM for large models: A 7B model typically needs 4 to 8 GB of RAM depending on quantization. Verify your server has enough memory beforehand. See AI PC for Beginners for hardware recommendations.
Automatic updates disrupting services: Occasionally a kernel update requires a reboot that interrupts running AI processes. Plan maintenance windows for kernel updates.

Hardware, Costs, and Security

Hardware

An AI server on Ubuntu requires at least 8 GB RAM and a CPU with four cores. For GPU-accelerated models, add an Nvidia graphics card with at least 8 GB VRAM. For CPU-only inference, a solid processor and ample RAM suffice.

Costs

Ubuntu Server is free. Costs come from hardware and power consumption. A modest mini PC with 16 GB RAM runs 200 to 400 euros. A server with a dedicated GPU ranges from 600 to 1500 euros depending on the model.

Security

A server in your own network isn’t automatically secure. Implement these measures:

Use SSH keys instead of passwords
Disable root login via SSH
Enable the UFW firewall and permit only necessary ports
Apply updates regularly, ideally automatically for security patches
Don’t expose the server directly to the internet; run it behind a reverse proxy or VPN instead

FAQ

Do I need the desktop version of Ubuntu for AI?

No, the server version is perfectly adequate and uses fewer resources. The command line is the standard for server operation. If you prefer a graphical interface, you can install the desktop version, but you’ll sacrifice memory and CPU performance in the process.

Which Ubuntu version should I install?

Use the current LTS release, which is Ubuntu 24.04 LTS. LTS versions receive five years of security updates and are more stable than the interim releases that ship every six months.

Can I run Ubuntu Server without a monitor?

Yes, that’s the typical setup. You install the server once with a monitor and keyboard connected, enable SSH, and then manage it entirely over the network afterward.

How much RAM do I need for local AI on Ubuntu?

Small models like Llama 3.2 with 3B parameters need 8 GB RAM. For 7B models, I’d recommend 16 GB. If you want to run multiple models simultaneously, plan for 32 GB or more.

Does Ollama work without a GPU?

Yes, Ollama runs on CPU, but it’s significantly slower. For experimentation, that’s fine. For regular use, a GPU is worth the investment.

How do I keep my Ubuntu Server up to date?

Run sudo apt update && sudo apt upgrade regularly. For security updates, configure unattended-upgrades to install them automatically.

Is Ubuntu Server secure enough for AI applications?

Yes, if you implement the basics: SSH keys, disabled root login, an active firewall, and regular updates. A server on your own network is far less exposed than one running in the cloud anyway.

Can I run multiple AI tools on Ubuntu at the same time?

Yes, you can install Ollama, Docker containers, and Python projects in parallel. Just ensure each service uses its own port and that you have enough hardware resources.

What should I do if Ollama won’t start?

Check the logs with journalctl -u ollama -e. Common causes are missing permissions, insufficient RAM, or a broken systemd service. For detailed troubleshooting, see the Ollama on Linux guide.

Do I need Docker for local AI on Ubuntu?

No, Docker is optional. Ollama and Python work fine without it. Docker becomes useful when you want to run multiple isolated AI environments or deploy applications more easily.

Can I access my Ubuntu Server from Windows?

Yes, on Windows use SSH clients like PuTTY or the built-in OpenSSH feature. You’ll need the server’s IP address and either your SSH key or password.

Sources

Ubuntu Documentation: https://documentation.ubuntu.com/
Ubuntu Server Download: https://ubuntu.com/download/server
Ollama Documentation: https://github.com/ollama/ollama
Docker Documentation: https://docs.docker.com/
UFW Documentation: https://help.ubuntu.com/community/UFW

Täglicher Artikel