AI Log Checker: Analyzing Linux Logs During Security Incidents (Commands + Automation)

IRC-Mania Die Seite rund um Sicherheit und Programmierung

Internet, Sicherheit , Internet Relay Chat

Gesponsert

What This Article Covers

• The Linux log landscape: which log file lives where (auth.log vs. secure, journalctl, audit.log, wtmp/btmp, bash_history).
• Command triage during a security incident: suspicious logins, failed logins, shell history, auditd, persistence (cron, systemd, ld.so.preload), SSH backdoor keys, /proc forensics.
• How to analyze logs locally with AI (Ollama, LogWhisperer, logllama, Sentinel) versus cloud services, and why local analysis is the right choice for security logs.
• Automated alerts: systemd timers + LLM + Discord/Slack/Telegram notifications.

Introduction

When you suspect a security incident, time is your enemy: whoever broke in leaves traces in logs, and the first question is always “which logs, which commands”. This article is your checklist plus the AI layer: how a local LLM can classify raw log lines and send automated alerts without your logs ever leaving your server.

The Linux Log Landscape: Where Each Log Lives

On pure systemd systems, the text files may not exist. Use: journalctl -u ssh -u sshd -S "2026-09-22 01:00:00" --no-pager instead.

Security Incident Triage: The Commands

1. Who successfully got in?

# Successful SSH logins (Debian/Ubuntu)
grep -E "Accepted (publickey|password)" /var/log/auth.log | tail -25

# Same on RHEL/Fedora
grep "Accepted" /var/log/secure | tail -25

# Who is currently logged in + recent logins
who
last -20
lastlog | grep -v "Never"

2. Failed Login Storms (Brute Force)

# Failed logins from btmp (not in text logs)
sudo lastb | head -20

# Most frequent attacker IPs
grep "Failed password" /var/log/auth.log | awk '{print $(NF-3)}' | sort | uniq -c | sort -rn | head

3. What Was Executed? Shell History + auditd

# History for the user (be aware: attackers often delete this)
cat /home/user/.bash_history | tail -50
cat /root/.bash_history | tail -50

# Better: auditd captures EVERY execution at the kernel level
sudo ausearch -m EXECVE --start recent | tail -100
sudo ausearch -i -m EXECVE --start today        # human-readable (-i = interpret)
sudo aureport --failed                          # failed events
sudo aureport -x --summary                      # all execve calls

auditd sees what history hides, because it sits in the kernel. Key rules to set up in advance (in /etc/audit/rules.d/):

-a always,exit -F arch=b64 -S execve -S execveat -k exec
-w /etc/passwd -p wa -k identity
-w /etc/shadow -p wa -k identity
-w /etc/sudoers -p wa -k privilege
-w /etc/cron.d/ -p wa -k persistence
-w /etc/systemd/system/ -p wa -k persistence
-w /root/.ssh/ -p wa -k persistence
-w /etc/ld.so.preload -p wa -k preload

4. Finding Persistence: Where the Attacker Hides “Day 2”

# All crontabs (hidden jobs)
for u in $(cut -d: -f1 /etc/passwd); do echo "== $u"; crontab -l -u $u 2>/dev/null; done
ls -la /etc/cron.d/ /etc/cron.daily/ /etc/cron.hourly/ /var/spool/cron/crontabs/

# systemd timers (modern cron disguise)
systemctl list-timers --all

# Newly created service files
ls -lt /etc/systemd/system/ /lib/systemd/system/ | head -15

# Shell startup scripts
grep -r "curl\|wget\|base64\|nc " /etc/profile.d/ ~/.bashrc ~/.profile /etc/profile 2>/dev/null

5. SSH Backdoor Keys

# Find all authorized_keys on the system
sudo find / -name "authorized_keys" -type f 2>/dev/null

# Check contents, any unknown keys?
cat /root/.ssh/authorized_keys /home/*/.ssh/authorized_keys 2>/dev/null

# Recently modified files system-wide (attacker artifacts)
sudo find / -ctime -2 -type f 2>/dev/null | grep -v -E "^/(proc|sys|run|dev|var/log)"

6. /proc Forensics: What Rootkits Can’t Hide

A userspace rootkit (LD_PRELOAD) can hide processes from ps. /proc is the source of truth:

# Binary of a suspicious process
sudo ls -l /proc/<PID>/exe

# Deleted but still running binaries (classic rootkit sign)
sudo ls -l /proc/*/exe 2>/dev/null | grep "(deleted)"

# Process environment and command line
sudo cat /proc/<PID>/cmdline | tr '\0' ' '; echo
sudo cat /proc/<PID>/environ | tr '\0' '\n' | grep -i "preload\|ld_"

# The classic preload rootkit check
cat /etc/ld.so.preload 2>/dev/null    # should be empty
ls -la /etc/ld.so.preload

7. Important to Know: Logs Are Tamperable

A serious attacker deletes auth.log lines and bash_history. Counter-evidence: auditd (kernel), wtmp/btmp (binary format, harder to edit), remote syslog (the copy on the central server can’t be forged by the attacker), and file timestamps. Always cross-check multiple sources.

AI Analysis of Logs: Local vs. Cloud

Cloud warning first: security logs contain IPs, usernames, hostnames, sometimes internal paths. That is exactly what an attacker needs. Pasting auth.log into ChatGPT or a web API is a privacy and security problem. That’s why local LLMs via Ollama are the right choice for log analysis. See Local AI and Ollama.

Ready-Made Local AI Log Tools

The Classic DIY Pipeline (Self-Hosted, Free)

The pattern behind all these tools is the same and buildable in about 50 lines:

# 1. Collect logs (here: recent auth.log failures)
grep "Failed password" /var/log/auth.log | tail -100 > /tmp/fails.txt

# 2. Send to local Ollama
ollama run llama3.1:8b "You are a Linux sysadmin. Analyze these failed logins:
Patterns? IPs? Brute force or targeted? Answer briefly and structured.
$(cat /tmp/fails.txt)"

# 3. Automate: systemd timer every 5 minutes
#    journalctl/Loki → Python → Ollama API → classification → alert

At production scale (this is exactly the LogWhisperer/Sentinel approach):

journalctl -f  (or Loki query every 5 min)
  → Keyword gate (ERROR/Failed/Accepted patterns only, no LLM needed)
  → Ollama LLM: "Noise or anomaly: [reason]?"
  → if anomaly → alert to Discord/Slack/Telegram/n8n webhook
  → dedup cache: don't report the same signature twice

Recommended local models: llama3.1:8b for security logs, gemma2:9b for general analysis (both around 5-6 GB RAM in Q4). n8n can build the same flow without Python. See Best n8n Workflows.

Important: The AI Classifies, You Decide

The LLM acts as a pre-filter (noise reduction, pattern explanation), not as forensic evidence. The actual investigation stays with the commands above. Good tools like logllama enforce “back every claim with the log line number,” and your prompt should demand the same: “Cite the line number as proof. No assumptions.”

Incident Response Checklist (In Order)

1. Who got in?    → grep "Accepted" auth.log/secure, last, lastlog
2. Who tried?     → sudo lastb, Failed-password-IPs
3. What ran?      → ausearch -m EXECVE --start recent, bash_history
4. What persists? → cron, systemd-timer, authorized_keys, ld.so.preload
5. What's running?→ /proc/*/exe (deleted), cmdline, environ
6. Secure it      → Revoke keys, reset passwords, remote syslog, auditd rules
7. Alert          → Run AI pipeline continuously, not just once

Further Reading

• IRC-Security.de: additional security articles and guides.
• IRC-Coding.de: programming tutorials and coding articles.
• Logging: logging strategy.
• Monitoring: alerting fundamentals.
• Agent Security: guardrails.
• Ollama: the local LLM behind the analysis.
• Best n8n Workflows: alert automation.
• Handling Secrets Properly: never ship credentials in logs.

Key Takeaways:

• Distro map: Debian/Ubuntu log auth to /var/log/auth.log, RHEL family to /var/log/secure, systemd fallback journalctl -u ssh.
• Incident commands: grep "Accepted", last/lastb, ausearch -m EXECVE --start recent, aureport --failed, crontab -l, systemctl list-timers, find / -name authorized_keys, /proc/PID/exe, /etc/ld.so.preload.
• Persistence check: cron (all users), systemd timers, startup scripts, SSH keys, preload.
• Local AI analysis via Ollama: LogWhisperer, logllama, log-explainer, Sentinel, or DIY journalctl → LLM → Discord/Slack/n8n alert. Never send logs to the cloud.
• The LLM is the pre-filter; the commands provide the evidence.

FAQ

Sources and Further Reading

• LogWhisperer, logllama, log-explainer
• Sentinel / OpenClaw Log-Anomaly, AI Log Analysis with Ollama
• auditd Incident Response, ausearch(8) man page
• IRC-Security.de: additional security articles.
• IRC-Coding.de: programming tutorials.

Premium VPS Hosting

Schnelle und sichere VPS Server

Gesponsert