01 - System Architecture

This document provides a complete mapping of the llmpowerhouse system architecture — every component, how they interconnect, and the design philosophies driving structural choices. It covers seven architectural layers from hardware through backup, plus networking, supporting apps, and the rationale behind major decisions.

The entire system runs on a single Mac Studio with model weights, documents, code, logs, and backups all stored on one external volume (/Volumes/AI_DATA). This portability lets the whole setup be rebuilt from scratch if the Mac itself is ever replaced. The architecture deliberately separates two front doors — one for human chat use, one for programmatic API access — while sharing knowledge banks between them for retrieval-augmented generation (RAG).

Mac Studio (Apple M3 Ultra, 28 cores, 96 GB unified memory)
  └── /Volumes/AI_DATA  (1.8 TB dedicated APFS volume)
        └── models/ollama/   (102 GB of model weights)
              └── Ollama (localhost:11434)  — the engine, OpenAI-compatible API

Three chat models and one embedding model are pulled into Ollama. Here is how they break down by size and role:

The Mac has 77.8 GiB of usable VRAM, configured for at most 2 resident models at once (OLLAMA_MAX_LOADED_MODELS=2). qwen3.6 + qwen3-vl together fit (40.5 GiB) and stay warm side by side. Either one paired with gpt-oss:120b exceeds the budget — selecting the big model evicts everything else, and switching back evicts it again, paying a multi-minute cold-load penalty each way.

In practice this means:

• qwen3.6 handles nearly all traffic.
• qwen3-vl only activates for image input.
• gpt-oss:120b is reserved for occasional hard problems where the load-time cost is acceptable.

                     ┌─────────────────────────────┐
                     │   Ollama :11434 (private)   │
                     │   never exposed to internet │
                     └───────────▲─────────────────┘
                                 │
              ┌──────────────────┴───────────────────┐
              │                                       │
   ┌──────────┴──────────┐                ┌───────────┴────────────┐
   │  Open WebUI :8080   │                │  agent-server :8788    │
   │  (human chat UI)    │                │  (API for apps/tools)  │
   │  + built-in RAG     │                │  + ChromaDB RAG        │
   │  no memory, no tools│                │  + per-user memory     │
   └─────────────────────┘                │  + MCP tool calling    │
                                          └───────────┬────────────┘
                                                      │
                                         ┌────────────┴─────────────┐
                                         │  gateway.py :8787 (legacy)│
                                         │  dumb proxy, ~20 old apps │
                                         └───────────────────────────┘

This is a deliberate design choice, not an unfinished migration. The two front doors serve different audiences with different capabilities.

gateway.py (port 8787) is the original, simple bridge — it forwards a request to Ollama and does nothing else. It still serves roughly 20 existing integrations and is explicitly not being retired.

agent-server (port 8788) is the actively developed system. It adds:

• Knowledge-base search
• Per-user memory
• MCP tool calling
• Real-time streaming
• Per-app API keys with scopes and rate limits

A key issued for one server does not work on the other — they have separate key stores. The :8443 vs :443 suffix (see networking) is the only visible difference from outside.

Open WebUI talks directly to Ollama, not through agent-server — so it has its own (simpler) retrieval but none of agent-server's memory or tool-calling.

knowledge-bank/
├── DOD-FM-Knowledge-Bank/     (financial-management documents, 288 MB)
├── K12-Knowledge-Bank/        (education standards/curricula, 1.8 GB)
├── Wiki/                      (curated one-page-per-concept notes, 257 pages)
├── _inbox/                    (drop zone for new files)
└── _index/
      └── chromadb/            (722 MB — the vector search index)

The foundation layer: documents are chunked (800 characters, 100-character overlap), embedded with nomic-embed-text, and stored in ChromaDB. A query is embedded the same way and matched by cosine similarity. This is described in the project's own documentation as "90% librarianship, 10% technology" — folder taxonomy, consistent filenames, and pruning stale duplicates matter more than any algorithm.

apps/llm-wiki/llm_wiki.py watches the knowledge banks and auto-drafts a distilled Markdown page per concept, which a human reviews before it's promoted. Wiki pages are themselves indexed by Tier 1, so a well-curated wiki page tends to out-compete raw source text in retrieval.

Designed but not yet deployed — reserved for if Tiers 1–2 demonstrably fail on multi-hop "what depends on what" questions, using LightRAG rather than a hand-rolled graph database.

A knowledge-bank/source_authority.json file nudges search results toward higher-trust documents (e.g., the actual regulation rather than a summary of it) without overriding relevance entirely.

This is the most architecturally interesting part of the system — a hand-built agent harness, not a framework. Full detail lives in 03_Agent_Harness_Deep_Dive.md; here's the shape:

Caller (app, curl, VS Code)
  │  POST /v1/chat/completions  +  Bearer API key
  ▼
auth.py            — validate key hash, scope, rate limit, quota
  ▼
agent_loop.py       — compose system prompt (AGENT.md + matched skill + memory)
  │                    check: is this a self-referential question? (skip tools if so)
  ▼
graph.py             — wave-based executor: run planner, fan out any tool calls
  │                    concurrently, fan back in, repeat until done or budget hit
  ├──► tools/registry.py  (hand-written tools: search_knowledge_base, get_study_plan)
  ├──► mcp_client.py      (external tools: filesystem, web search, PDF reader, planning)
  └──► hooks/              (pre-tool: block disallowed writes; post-tool: trigger re-sync)
  ▼
Ollama :11434         — the actual model call
  ▼
memory_pipeline.py   — record what was asked/found (fire-and-forget, doesn't block response)
  ▼
Response to caller (streamed or complete)

Every request, tool call, skill match, and feedback signal is logged to logs/agents/*.jsonl — this audit trail is what enabled the July 30 latency bug (see 05_Gap_Analysis.md) to be diagnosed from evidence instead of guesswork.

Internet
  │
  ▼
Tailscale Funnel (encrypted tunnel, no router port-forwarding)
├── :443   →  gateway.py     (legacy, ~20 apps)
└── :8443  →  agent-server   (current)

Ollama itself is never exposed beyond localhost and the private Tailscale network — it has no authentication of its own, so this is treated as an absolute rule, not a preference. The only public doors are the two proxies above, both of which enforce their own API keys before anything reaches Ollama.

Admin access to the Mac itself is tiered: browser-based admin panels first, SSH only when needed, screen sharing (VNC) as a last resort.

Backup-AI.command  →  backups/<timestamp>/
    (keys.db, memory.db, AGENT.md, skills/, *.command scripts — small, irreplaceable)
Backup-AI.command full  →  also copies the 722 MB ChromaDB index

Model weights (104 GB) and source PDFs are deliberately not backed up in the same way — they're reproducible (ollama pull, re-download) rather than irreplaceable.

Open risk: Backups currently live on the same volume they protect, which covers accidental deletion but not drive failure. See 05_Gap_Analysis.md.

Every major structural choice traces back to one of two constraints:

1. The 96 GB memory ceiling — which forced the "workhorse default, big model for hard cases" model policy and the two-server-not-one exposure design.
2. A philosophy of layering cheap deterministic mechanisms in front of expensive flexible ones:
   • Regex before embeddings before a full model call
   • Vector RAG before wiki before graph
   • Hand-written tools and MCP tools sharing one registry so the routing logic never needs to know which is which

That pattern is worth keeping in mind when extending the system — see 08_Extending_The_Harness_Agent.md.