Running AI Agents 24/7

IRC-Mania Die Seite rund um Sicherheit und Programmierung

Internet, Sicherheit , Internet Relay Chat

Gesponsert

What this article covers

• How to run AI agents 24/7.
• Monitoring, maintenance, and error handling for continuous operation.
• How to keep agents reliable and stable.
• Real-world examples of continuous-operation agents.
• Best practices for availability, performance, and maintenance.

Introduction: Understanding continuous operation

Continuous operation means your agent runs 24/7, processes tasks without interruption, monitors itself, and recovers from errors automatically. It’s not just “start the agent once,” but rather “the agent runs reliably all the time.”

This article is for anyone who needs to run agents continuously. For foundational concepts, see Running AI agents locally and Monitoring.

Why do I need continuous operation?

Imagine your agent processes emails around the clock. Not just when you start it, but all the time: overnight, on weekends, automatically restarting if something fails. Continuous operation means reliable, uninterrupted availability.

Continuous operation in a nutshell

Agent runs 24/7 → Monitoring watches it → On error: auto-restart → On success: continue working. Logging, alerting, and self-healing ensure reliability.

The core idea: sustained availability without manual intervention.

Who is this article for?

• Production teams running agents continuously.
• DevOps responsible for uptime.
• Enterprises needing 24/7 automation.
• Self-hosters running reliable agents.

Key concepts

• Monitoring - System oversight. Useful for: continuous operation.
• Logging - Event recording. Useful for: debugging.
• Ollama - Model server. Useful for: continuous operation.
• Docker - Deployment. Useful for: auto-restart.
• Systemd - Service management. Useful for: auto-start.

Architecture for continuous operation

Agent runs (24/7)
    │
    ├─ Monitoring: health checks every 30s
    ├─ Logging: record all actions
    ├─ Alerting: notify on failures
    └─ Self-healing: restart on error
    │
    ▼
Systemd / Docker restart policy
    │
    ├─ On crash: auto-restart
    ├─ On OOM: restart with more memory
    └─ On hang: timeout → restart

Setup: Systemd for continuous operation

# /etc/systemd/system/ki-agent.service
[Unit]
Description=KI-Agent
After=network.target ollama.service

[Service]
Type=simple
User=ki-agent
WorkingDirectory=/opt/ki-agent
ExecStart=/usr/bin/python3 /opt/ki-agent/agent.py
Restart=always
RestartSec=10
StandardOutput=journal
StandardError=journal

# Resource limits
MemoryMax=4G
CPUQuota=80%

[Install]
WantedBy=multi-user.target

# Enable and start
sudo systemctl enable ki-agent
sudo systemctl start ki-agent

Setup: Docker for continuous operation

version: "3.8"

services:
  agent:
    build: ./agent
    restart: always  # auto-restart on failure
    environment:
      - OLLAMA_URL=http://ollama:11434
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:8000/health"]
      interval: 30s
      timeout: 10s
      retries: 3
      start_period: 40s
    deploy:
      resources:
        limits:
          memory: 4G
          cpus: '2.0'
        reservations:
          memory: 2G
          cpus: '1.0'
    networks:
      - agent_net

  ollama:
    image: ollama/ollama:latest
    restart: always
    volumes:
      - ollama_data:/root/.ollama
    networks:
      - agent_net

volumes:
  ollama_data:

networks:
  agent_net:
    driver: bridge

Monitoring and alerting

import time
import logging
from datetime import datetime

class AgentMonitor:
    """Monitoring for agents"""

    def __init__(self):
        self.logger = logging.getLogger("agent")
        self.error_count = 0
        self.last_success = datetime.now()

    def check_health(self):
        """Health check"""
        try:
            # Test agent
            response = requests.get("http://localhost:8000/health", timeout=5)
            if response.status_code == 200:
                self.last_success = datetime.now()
                self.error_count = 0
                return True
        except Exception as e:
            self.error_count += 1
            self.logger.error(f"Health check failed: {e}")

            if self.error_count > 3:
                self.alert("Agent unhealthy", e)
                return False

    def alert(self, message, error):
        """Send alert"""
        # Email, Slack, PagerDuty, etc.
        send_notification(f"ALERT: {message}\nError: {error}")

    def run(self):
        """Monitoring loop"""
        while True:
            self.check_health()
            time.sleep(30)  # Every 30 seconds

Error handling

class ResilientAgent:
    """Agent with error handling"""

    async def run_with_retry(self, task, max_retries=3):
        """Task with retry"""
        for attempt in range(max_retries):
            try:
                result = await self.process(task)
                return result
            except Exception as e:
                self.logger.warning(f"Attempt {attempt + 1} failed: {e}")

                if attempt == max_retries - 1:
                    self.alert(f"Task failed after {max_retries} attempts", e)
                    raise

                await asyncio.sleep(2 ** attempt)  # Exponential backoff

    async def process(self, task):
        """Process task"""
        try:
            # Agent logic
            result = await self.execute(task)
            return result
        except OllamaError as e:
            # Ollama error: reload model?
            await self.handle_ollama_error(e)
            raise
        except MemoryError as e:
            # OOM: use less context?
            await self.handle_oom(e)
            raise

Security considerations

• Monitoring: The agent should monitor itself. Alert on anomalies.
• Resource limits: Memory and CPU limits prevent the agent from consuming all resources.
• Auto-restart: Restart automatically on failure, but not infinitely.
• Logging: Record all actions for debugging and audit. See Logging.
• Backup: Regularly back up configuration and data. See Backup.

Common pitfalls

• No monitoring: Without monitoring, you won’t know when the agent fails.
• Memory leaks: Long-running agents can develop memory leaks. Monitoring is essential.
• No auto-restart: When the agent crashes, it stays down. Configure auto-restart.
• Restart loops: Endless restarts on persistent errors exhaust resources. Use max retries or circuit-breaker patterns.
• Missing alerts: You should be notified of failures immediately, not discover them hours later.

Further reading

• Monitoring - System oversight.
• Logging - Event recording.
• Docker - Deployment.
• Running AI agents locally - Overview.
• Backup - Data protection.
• Audit Logging - Audit trail.

Key takeaways:

• Continuous operation: agent runs 24/7, monitors itself, recovers automatically.
• Monitoring: health checks, logging, alerting.
• Auto-restart: Systemd or Docker with restart=always.
• Resource limits prevent out-of-memory crashes.
• For production: monitoring and alerting are mandatory.

FAQ

Sources and further reading

• Systemd - Service management.
• Docker - Containers.
• Ollama - Model server.

Premium VPS Hosting

Schnelle und sichere VPS Server

Gesponsert