Your prompts are the most honest diary you keep. Every cloud chatbot gets to read it. Here's how to run a private AI on hardware you already own — Ollama for the models, Open WebUI for the pretty face — and keep your data off someone else's server.

Why your AI should live at home

Operators self-host for three reasons: privacy, cost, and control.

Privacy. Pasting production logs, config files, and half-written code into a cloud chat window ships your infrastructure's internals to a third party. A local model never phones home. For anyone who debugs systems for a living, that's not paranoia — it's hygiene.

Cost. API tokens meter every thought. A local setup costs whatever your hardware already cost, plus electricity. If you chat with AI daily, the math gets embarrassing for the cloud fast.

Control. No policy changes, no model deprecations, no "we've updated our terms" surprises. Your model, your rules. And it works offline — on a plane, in a basement lab, behind an air gap.

The honest trade-off: open-weight models trail frontier cloud models on the hardest reasoning tasks. But for daily work — drafting, summarizing logs, scripting help, rubber-duck debugging — a good 8B or 14B model is plenty. Self-hosting isn't about beating the frontier. It's about owning the 80%.

The VRAM math nobody skips

Before pulling anything, know what fits. Rule of thumb for a quantized model: roughly 0.5 GB of VRAM per billion parameters at Q4 quantization, plus a few GB of headroom for context.

  • 3B → ~2 GB: runs on almost anything, even a CPU-only laptop. Fast, a bit dumb, great for quick drafts.
  • 7–8B → ~5–6 GB: the sweet spot for most desktops. Handles coding help, summaries, most daily tasks.
  • 13–14B → ~9–10 GB: noticeably smarter; wants a midrange GPU or generous system RAM.
  • 27–32B → ~18–20 GB: the ceiling for high-end consumer GPUs.
  • 70B → ~40 GB: workstation territory. Skip it unless you own the iron.

CPU inference is genuinely usable at 3B and tolerable at 8B. NVIDIA GPUs just work; AMD users get ROCm support out of the box these days. Start smaller than your ego wants — you can always pull a bigger model later, and a fast small model you actually use beats a slow big one you abandon.

Step 1: Install Ollama

One script, one daemon:

curl -fsSL https://ollama.com/install.sh | sh
sudo systemctl enable --now ollama

Yes, that's a pipe to shell. If that makes your skin crawl, the script is short and readable — audit it first, then run. Operator's choice either way.

Verify the daemon is up:

curl -s http://127.0.0.1:11434/api/version
ollama list

Ollama listens on 127.0.0.1:11434 — localhost only. Keep it that way until you have both a reason and authentication to change it. An unauthenticated model server on your LAN is a gift to everyone on the network.

Pull your first model and take it for a spin:

ollama pull llama3.2:3b
ollama run llama3.2:3b

For coding help, qwen2.5-coder:7b punches well above its weight. For general chat, llama3.2:8b or qwen2.5:7b are solid defaults. The full menu lives in the model library, and ollama pull grabs a sensible quantization for your hardware automatically.

Step 2: Give it a face with Open WebUI

The terminal is fine for you. Your family, your team, and your phone want a web UI. Open WebUI is the ChatGPT-shaped front end that talks to Ollama:

docker run -d -p 3000:8080 \
  --add-host=host.docker.internal:host-gateway \
  -v open-webui:/app/backend/data \
  --name open-webui --restart always \
  ghcr.io/open-webui/open-webui:main

Open http://localhost:3000 and create the first account — it becomes the admin. The --add-host=host.docker.internal:host-gateway flag is the one line everyone forgets; it's what lets the container reach Ollama running on your host.

What you get for that one command: mid-conversation model switching, searchable conversation history, reusable prompt templates, voice input, and user accounts with roles if you share the box. It also speaks to any OpenAI-compatible backend, so you're not locked to Ollama forever.

Step 3: Chat with your own documents

This is the feature that sells self-hosting. Open WebUI's RAG (retrieval-augmented generation) lets you upload PDFs, docs, and text files, then ask questions grounded in your material — runbooks, vendor manuals, that 200-page PDF nobody has ever fully read.

Upload a document, ask about it, and the answers cite your files instead of hallucinating. It turns the thing from a chatbot into a junior engineer who's actually read the docs. For a homelab, point it at your own notes and watch it become the most useful member of the household.

The gotchas, learned the hard way

Don't expose it naked. The moment you bind port 3000 to 0.0.0.0, it's on your LAN. If it ever leaves your machine, put it behind a reverse proxy with auth — or better, a VPN like Tailscale or WireGuard. In Open WebUI, disable open registration or require admin approval for new signups.

Keep models fresh. ollama pull <model> re-pulls and updates. Models get quiet revisions; a stale pull is a stale brain.

Respect the OOM killer. Loading a 14B model on a box with 12 GB of RAM ends exactly how you'd expect. Match the model to the hardware — see the math above.

Secrets still matter. Local doesn't mean careless. Prompts land in logs and conversation history, so treat the chat like any other system holding your data.

Back up the brain. Your chats, users, and uploaded documents live in the open-webui Docker volume. Back it up like anything else stateful — it's the one piece you'd actually miss.

When the cloud still wins

Honesty check, because someone has to say it: for frontier reasoning, giant context windows, or heavy agentic coding sessions, the big cloud models are still ahead — and a pay-per-token API is cheaper than a GPU farm you'll never fully utilize. Plenty of operators run both: local by default for the daily 80%, cloud for the hard 20%. That's not a compromise. That's architecture.

Sources