Web Chatbots with Local AI

What this article covers

• How a web chatbot with local AI works
• What components a chatbot interface needs
• How frontend, backend, and language model interact
• How to integrate RAG and document search into your web chatbot
• Privacy, performance, and common pitfalls

Introduction: Web chatbots with local AI

A web chatbot on your own website is now standard for many businesses. It answers questions, guides users, and takes pressure off support teams. When the AI runs locally, all conversations and data stay within your own network. This is particularly valuable for privacy-sensitive industries.

A local web chatbot has several components: the visible chat interface in the browser, a backend that receives requests, and the language model with an optional knowledge base. Understanding these pieces lets you build your own chatbot or customize an existing tool.

Why use a local web chatbot?

Cloud-based chatbots are easy to integrate, but they send user queries to external providers. For many businesses, that’s a deal-breaker. Local chatbots offer:

• Privacy: No transmission of personal data
• Control: Your own model, your own content, your own logs
• Cost: No per-message fees
• Customization: Brand voice, specialized knowledge, tailored responses
• Independence: No dependency on external APIs

How web chatbots work

Here’s a typical flow:

1. User types in chat window: A JavaScript component on the webpage captures input
2. Request to backend: Frontend sends the message to a local server
3. RAG retrieval (optional): Backend searches for relevant document chunks from the knowledge base
4. Build prompt: Language model receives question and context
5. Generate response: Model produces an answer
6. Display response: Backend returns answer to frontend, which displays it

Key terms:

• Frontend: The visible chat interface in the browser
• Backend: Server that mediates between frontend and AI
• API: Interface for exchanging requests and responses
• Streaming: Response is delivered piece by piece, not all at once
• Session: Conversation history that the model keeps track of
• RAG: Knowledge base for fact-based answers

Who should use a web chatbot?

• Website operators who want to automate user questions
• Organizations with data protection requirements
• Support teams looking to handle frequent inquiries automatically
• Developers building custom chat solutions

Key terms for web chatbots

• Open WebUI: Web chat interface for Ollama
• AnythingLLM: RAG platform with chat interface
• FastAPI: Python framework for a simple backend
• WebSocket: Protocol for fast, bidirectional communication
• SSE: Server-Sent Events, commonly used for streaming responses
• CORS: Security mechanism that determines which websites can access your API

Frontend: The chat interface

The frontend typically consists of HTML, CSS, and JavaScript. It displays a chatbox, input field, and message history. A basic example:

<div id="chat-container">
  <div id="messages"></div>
  <input type="text" id="user-input" placeholder="Ask a question...">
  <button onclick="sendMessage()">Send</button>
</div>

The JavaScript sends the message:

async function sendMessage() {
  const input = document.getElementById('user-input');
  const text = input.value;
  input.value = '';

  const response = await fetch('/api/chat', {
    method: 'POST',
    headers: { 'Content-Type': 'application/json' },
    body: JSON.stringify({ message: text })
  });

  const data = await response.json();
  document.getElementById('messages').innerHTML += `<p>${data.reply}</p>`;
}

For better user experience, implement streaming so responses appear word by word instead of all at once after generation completes.

Backend: API between browser and AI

A simple FastAPI backend:

from fastapi import FastAPI
from fastapi.middleware.cors import CORSMiddleware
import requests

app = FastAPI()

app.add_middleware(
    CORSMiddleware,
    allow_origins=["*"],
    allow_methods=["*"],
    allow_headers=["*"],
)

@app.post("/api/chat")
async def chat(payload: dict):
    frage = payload["message"]
    antwort = requests.post(
        "http://localhost:11434/api/generate",
        json={
            "model": "llama3.1:8b",
            "prompt": frage,
            "stream": False
        }
    ).json()
    return {"reply": antwort["response"]}

You can extend the backend with RAG, session management, user authentication, and rate limiting.

RAG in your web chatbot

To provide fact-based answers, add a knowledge base:

1. Load and chunk documents
2. Generate embeddings and store them in Chroma or Qdrant
3. Search for matching chunks on each request
4. Include chunks and question together in the prompt

Example:

Here are excerpts from the documentation:
{chunks}

Answer the following question based only on these excerpts:
{frage}

Security and privacy

• Configure CORS correctly: Not every website should access your API
• Enforce HTTPS: Encrypted connection prevents eavesdropping
• Authentication: Use sessions or API keys for protected access
• Rate limiting: Prevents abuse
• Minimize logging: Store chat histories only when necessary and in GDPR-compliant ways

Common pitfalls with web chatbots

• CORS errors: Frontend and backend can’t simply work together without proper configuration
• No streaming: Users wait a long time for responses
• Session loss: Without persistence, the bot forgets context
• RAG quality: Poor chunks or wrong embedding models produce poor answers
• Too many concurrent users: Local hardware has limits on parallel requests

Further reading and resources

• BotServ.de Open WebUI
• BotServ.de AnythingLLM with Docker
• BotServ.de Local RAG
• FastAPI
• Open WebUI

FAQ: Web chatbots with local AI

Do I need programming skills? For simple integrations with Open WebUI, minimal skills suffice. Custom solutions require more experience.

Is a local web chatbot slow? Depending on model and hardware, it may take longer than cloud providers. Modern GPUs and smaller models deliver acceptable speeds.

Can I embed the chatbot on a subpage? Yes, the frontend integrates into any webpage.

How do I update the chatbot’s knowledge? Re-read documents into the vector database or update chunks.

Is local operation suitable for customer websites? Yes, if privacy and availability are guaranteed. For high load, you can supplement with a cloud server.

Sources and further reading

• FastAPI: https://fastapi.tiangolo.com/
• Open WebUI: https://openwebui.com/
• Ollama API: https://ollama.com/
• AnythingLLM: https://useanything.com/

Summary: Web chatbots with local AI

A web chatbot with local AI consists of frontend, backend, and language model. RAG enables fact-based answers from your own documents. Local operation protects data and cuts ongoing costs. Proper CORS configuration, HTTPS, streaming, and thoughtful session management are essential. Master these components and you can run your own privacy-compliant website assistant.