02 - The Build Process, Step by Step

This document traces the sequential build of a local macOS AI infrastructure from bare-metal prep through custom agent harness development. It maps each installation phase to the specific operational requirements it solved, detailing model selection, retrieval architecture, routing logic, and the iterative debugging discipline that stabilized the system as it scaled from a single endpoint to a multi-user service.

Prerequisite prep happens before any AI software installs. Sleep must be disabled so the server does not interrupt mid-request, followed by installing Homebrew and a lightweight toolchain (git, python@3.12, ffmpeg, htop, jq, wget). An SSH key is generated for GitHub, and a dedicated external volume (/Volumes/AI_DATA) is provisioned separately from the boot disk. This physical separation later enables a "rebuild on a new Mac" workflow that takes half a day rather than requiring a full reinstall: simply redo Phases 0–1, ollama pull the models again, and restore the external data volume. A day-zero snapshot of brew list, system_profiler, and df -h was saved to backups/ as inexpensive insurance for later dependency audits.

Ollama is installed via Homebrew, but the storage redirect must happen before pulling any model:

export OLLAMA_MODELS=/Volumes/AI_DATA/models/ollama

Pulling models before the redirect causes them to consume boot disk space. The first model pulled was the large reasoning model (gpt-oss:120b, ~64 GB, taking 30–90 minutes). Ollama exposes an OpenAI-compatible API at localhost:11434/v1 — this single endpoint becomes the call target for every subsequent layer (Open WebUI, agent-server, and ~19 other integrated projects).

Two additional models were added to round out the inference pipeline:

• qwen3.6:35b-a3b: Selected as the fast daily-driver once it became clear the large reasoning model alone was too slow for routine tasks.
• qwen3-vl:30b: Paired with Whisper (large-v3-turbo) for vision and speech-to-text. Since Ollama only passes still images to Qwen3-VL (no native video support), an ffmpeg frame-extraction workaround was documented for video input.

LM Studio was configured as an optional side tool for prototyping new models via the MLX runtime (~10–20% faster on Apple Silicon). This established a standing upgrade pattern: trial in LM Studio, promote the winner to Ollama.

Open WebUI (a ChatGPT-style interface) was deployed inside its own Python virtual environment with storage directed to the AI_DATA volume. It was exposed on the LAN via --host 0.0.0.0 (explicitly bound to all interfaces, not a single IP) and accessed remotely through Tailscale. A one-click start-ai-stack.command replaced manual terminal workflows, using nohup and pgrep guards to prevent duplicate processes on double-clicks.

Real-world bug: Newly pulled models default to Private visibility in Open WebUI, causing non-admin users to see an empty model list until visibility was explicitly set to Public in the admin panel.

Knowledge management followed a three-tier architecture from KNOWLEDGE_MANAGEMENT_GUIDE.md:

1. Tier 1 (Populate): Scripts fetched two document libraries — DOD-FM (financial-management regulations) and K-12 (education standards/curricula) — into a fixed folder taxonomy (01-Regulations/, 02-DoD-Guidance/, etc.) with numbered prefixes. Silent failure fix: Early scripts saved HTML error pages as .pdf files, which passed size-only validity checks. The patched fix-broken-pdfs.sh introduced a magic-byte check (%PDF header) to catch corruptions.
2. Tier 2 (Automate): apps/llm-wiki/llm_wiki.py paired with a launchd file-watcher auto-drafts wiki pages within seconds of document arrival, always pending human review before promotion.
3. Tier 3 (Graph RAG): Deliberately unbuilt until a real test-question failure demonstrates the overhead is warranted.

Cross-bank leakage bug: DOD-FM content appeared in K-12 chats due to Open WebUI's model-level knowledge attachment. Fixed by implementing a custom filter function, knowledge_scope_filter.py.

Wiring external projects to the local AI follows a strict pattern: point the project's OpenAI-client configuration at Ollama's endpoint (http://localhost:11434/v1 or the Tailscale hostname), use any string as the API key (Ollama does not validate it), and specify a model name. A routing helper (ai_chain.py) was built to auto-select the vision model for image input, default to the fast workhorse for standard text, and escalate only to the large reasoning model for genuinely difficult, multi-step, or compliance-sensitive requests. VS Code was integrated via the Continue extension pointed at the same Ollama endpoint.

To allow a showcase site to demonstrate the system publicly without exposing Ollama directly, a lightweight proxy (gateway.py) was built behind Tailscale Funnel. It validates a per-app API key and model allow-list, applies rate limiting, and only then forwards traffic to Ollama. This was later updated so public demo routes through Open WebUI instead of bare Ollama, ensuring retrieval pipelines and the leakage filter apply to external traffic as well.

With retrieval and the model stack stabilized, multi-user access required:

• LAN discovery via the Mac's .local hostname
• Remote access via Tailscale (free tier, supporting several concurrent users)
• A three-layer admin-access model: browser admin first, SSH for diagnostics, screen sharing strictly reserved as a last resort

The workhorse model was set as the shared default; the large reasoning model was restricted to an admin-only path. Serving the larger model to multiple concurrent users exceeded the memory budget.

A two-stage backup strategy was implemented: an interim phase of inventory snapshots plus off-machine copies of irreplaceables, followed by a full plan after acquiring a 2 TB external SSD. Time Machine was configured specifically for Apple Silicon, with model weights and virtual environments deliberately excluded (they are reproducible via scripts). A strict requirement was enforced: test-restore at least one file manually before trusting the setup.

This phase marks the project's center of gravity shifting from configuring third-party software to building a custom Python service. The original Phase 7 placeholder was split out into AGENT_SERVER_GUIDE_v2.md and constructed as a numbered sequence:

• Step 1 — Bare passthrough: A minimal FastAPI server proxying to Ollama, proving the OpenAI-compatible contract works end-to-end.
• Step 1.5 — Real authentication: Hashed API keys (never stored raw), scopes (chat/tools/admin), per-key rate limits and daily token quotas, plus an audit log.
• Step 2 — Retrieval as a callable tool: search_knowledge_base allows the model to iteratively re-query if its first retrieval was weak, instead of being stuck with whatever was statically pasted into the prompt.
• Step 2.5 — Orchestration engine: An earlier draft proved too rigid; a wave-based async task graph (graph.py) was built from scratch to fan out concurrent tool calls and automatically fan them back in per request step.
• Step 3 / 3.5 — Behavior configuration: A keyword-matched skills system (skills/*.md) and a base system prompt (AGENT.md) defining identity, tool-use policy, and guardrails. Files are read fresh per request so edits take effect immediately without restarts.
• Step 3.6 — Memory: A SQLite-backed per-user profile (grade level, subject focus, knowledge gaps) that only updates from explicit user input, never from retrieved documents or model output.
• Step 4 / 4.5 / 4.6 — Tooling: An MCP client for external tool integration (filesystem, web search, PDF reading, planning), source-authority re-ranking to fix ranking anomalies where lower-quality documents outran actual regulations, and retrieval depth tuning.
• Step 5 — Guardrails as code: A hook system blocking file writes outside the knowledge-bank folder pre-execution, and triggering background re-indexing post-write.
• Step 5.5 / 5.6 — Measurement: A feedback endpoint, generated eval dashboard, and LLM-as-judge grading mode for test suites (deliberately using a different judging model to avoid self-preference bias).
• Step 5.7 — Latency optimization: Fixed Ollama's streaming format mismatch with plain JSON reads to enable real token-by-token streaming, paired with a pooled HTTP client and initial production latency baselines.
• Step 5.9 / 5.10 — Routing fix: A traced 14-second latency bug (the model hallucinating file searches instead of answering from its own prompt) required a two-tier gate: a fast regex check followed by an embedding-similarity check, both calibrated against logged production data. Full narrative in AGENT_LATENCY_INVESTIGATION_SUMMARY.md.
• Step 6 — Public deployment: A second Tailscale Funnel port exposes the agent-server (:8443) alongside the legacy gateway (:443).
• Step 7 — Fine-tuning (partial): A dataset-harvesting script isolates graded-PASS question/answer pairs; a LoRA training run completed successfully on a small substitute model as a pipeline smoke test.
• Step 8 — Guardrails, restated: Read-only by default, writes confined to sandboxed folders, full audit logging, and zero privileged credentials (Tailscale admin, backup drives) exposed to the agent loop.

Notice what remains consistent: each phase produced a immediately runnable increment, real usage exposed specific failure modes, and fixes targeted the narrowest mechanism that actually resolved the root cause (a magic-byte validation check instead of a pipeline rewrite; a regex gate before an embedding gate, rather than routing everything to the model). Ship, observe, fix precisely, write it down. That build discipline — independent of the specific tools selected — is the most transferable outcome of this project. Reference 04_Cheat_Sheet.md for operational commands and 08_Extending_The_Harness_Agent.md for subsequent expansion paths.

• Physical separation drives rebuildability: Keeping AI runtime, models, and external data on independent volumes enables rapid Mac swaps without full reinstallation cycles.
• Model routing prevents bottlenecks: Trial-proving on lightweight runtimes (LM Studio/MLX) before committing to Ollama, combined with strict workhorse/reasoning model scaling, keeps latency predictable under load.
• Retrieval leaks require scope guards: Cross-knowledge bank contamination and silent PDF corruptions were solved through targeted visibility filters and magic-byte validation rather than architectural overhauls.
• Guardrails scale via hooks, not policies: Baking read/write restrictions, audit logging, and credential isolation directly into pre/post-tool execution hooks enforces security at runtime without manual oversight.
• Observation beats assumption: Routing fixes, latency tuning, and storage redirects were all resolved by tracing production logs first, then applying measured, narrow interventions.