00 - Local Server "llmpowerhouse" Timeline

The llmpowerhouse project represents a fully local, DIY AI infrastructure built on a Mac Studio over one week in July 2026. Spanning hardware initialization to a fully operational agent server with RAG, memory, and tool use, this timeline tracks the exact dates, technical decisions, and iterative problem-solving that turned bare metal into a working knowledge engine.

Roughly seven days turned a blank Mac into a working local AI server with retrieval-augmented generation, persistent memory, tool integration, and a public demonstration. Zero components were procured as managed cloud services; every piece runs entirely on the owner's hardware.

Before any AI software was installed, the project captured a comprehensive baseline: backups/hardware_info.txt (Mac Studio, Apple M3 Ultra, 28 cores, 96 GB RAM), backups/brew_packages.txt (git, python@3.12, ffmpeg, htop, jq, wget, VS Code, and codec libraries), and backups/disk_layout.txt (a dedicated 1.8 TB /Volumes/AI_DATA APFS volume isolated from the boot disk to ensure the entire system remains "one folder: portable, backupable, rebuildable").

Two structural documents were drafted before a single model was pulled: MAC_BASICS_CHEATSHEET.md (a terminal/shortcut reference for newcomers) and KNOWLEDGE_MANAGEMENT_GUIDE.md (outlining a three-tier RAG/Wiki/Graph strategy before knowledge existed). This "document-the-plan-before-building" discipline became a recurring pattern. The work mapped directly to Phase 0 of AI_SERVER_SETUP_GUIDE.md: macOS updates, disabled sleep mode, Homebrew/dev tools installation, SSH key generation for GitHub, and the initial AI_DATA folder structure (models/, knowledge-bank/, apps/, logs/, backups/).

Phase 1 & 2 (Ollama + Model Suite): Ollama was installed with weights directed to /Volumes/AI_DATA/models/ollama to keep model files off the boot disk. Three models were pulled, totaling 102 GB: gpt-oss:120b (~64 GB, primary deep reasoner/"MAIN"), qwen3.6:35b-a3b (~22 GB, fast mixture-of-experts daily driver), and qwen3-vl:30b (~18 GB, sole vision-capable model). Whisper (large-v3-turbo) was added for speech-to-text. Vision and audio were split across separate models due to macOS audio limitations with combined "omni" architectures at the time.

Phase 3 (Interface & Networking): Open WebUI was deployed as a ChatGPT-style interface within a dedicated Python virtual environment, exposed via LAN (llmpowerhouse.local:8080) and remotely through Tailscale. The companion showcase site apps/llmpowerhouse-site/ was built using Next.js/TypeScript/Tailwind, featuring an interactive setup checklist, roadmap, and a "Live Demo" bridging directly to the home server. Its git history shows 18 commits on 2026-07-25 alone, rapidly patching cold-load timeouts, history-log bugs, gateway process-kill patterns, and desktop Start/Stop/Restart buttons after rate limits were removed.

A gateway.py bridge (port 8787) was engineered to mediate between the public internet (via Tailscale Funnel) and Ollama, enforcing per-app API keys for isolated credential revocation. Finalized restart-ai-stack.command, stop-ai-stack.command, and initial knowledge-bank download scripts (download-omb-dfas-service-docs.sh, download-sap-oracle-platform-docs.sh, download-gao-audit-service-reports.sh, download-crs-fm-acquisition-reports.sh), establishing the foundational DOD-FM financial-management library.

Ingestion machinery expanded rapidly with four additional download scripts for a K12 education standards bank (AMC/AP/SAT/PSAT materials, state curricula, elite-school catalogs). A critical validation bug was discovered and patched: some "downloaded" files were actually HTML error pages saved with a .pdf extension, passing a naive size-only validity check. Post-audit corrections revealed accurate counts: K12 (280 valid / 15 broken / 3 unobtainable) and DOD-FM (58 valid / 0 broken / 26 unobtainable).

SYSTEM_OVERVIEW_SUMMARY.md was documented as a one-page architectural map that remains the primary orientation guide. A key operational constraint was noted: the AI's sandbox cannot make outbound requests (403 blocked-by-allowlist), necessitating manual execution of all download scripts by the user in their local Terminal.

Tier 2 of the knowledge strategy was automated: apps/llm-wiki/llm_wiki.py was deployed alongside a launchd-managed file watcher (com.llmpowerhouse.wiki-watch.plist) to auto-draft wiki pages within ~10 seconds of new document ingestion, pending human review. Same-day crash fixes addressed launchd's stripped PATH hiding the fswatch binary, bare TimeoutError exceptions halting runs, stdout buffering delaying progress feedback, and unsanitized filenames containing /. The day closed with 257 live wiki pages (75 DOD-FM, 182 K12).

A cross-contamination bug was traced and resolved where DOD-FM content leaked into K-12 chats. Root cause analysis revealed Open WebUI's knowledge attachment was scoped to the model level rather than per-conversation. The fix involved deploying a custom Open WebUI "inlet filter" (apps/open-webui-filters/knowledge_scope_filter.py) as a deterministic second line of defense, paired with test-questions.md (10 real questions per bank) to convert subjective validation into a pass/fail checklist.

This marked the strategic shift from configuring tools to building a purpose-built agent. The apps/agent-server/ directory was initialized with commercial-grade scaffolding: keys_admin.py, keystore.py, and auth.py (hashed keys, scopes, rate limits, quotas), graph.py (a custom async task-graph engine replacing naive point-to-point wiring), and kb_ingest.py/kb_sync.py (ChromaDB-backed retrieval supplanting Open WebUI's native RAG). The architecture was codified in AGENT_SERVER_GUIDE_v2.md, superseding the original "Phase 7" placeholder.

Nearly the entire agent framework was engineered in a single development cycle:

• Ran an initial 20-question quality baseline (13 PASS / 7 FAIL, 65%).
• Implemented the skills system (skills/loader.py with keyword-matched Markdown rule files) and base system prompt (AGENT.md).
• Developed a Model Context Protocol (MCP) client alongside a curated catalog of candidate servers.
• Added source-authority re-ranking to correct regulation precedence issues where lower-authority documents out-ranked primary regulations.
• Layered critical safeguards: write-path guards restricting file operations to the knowledge-bank directory, auto-resync triggers post-write, and a per-user memory backend (memory_store.py, SQLite-backed tracking grade level, subjects, and knowledge gaps).
• Added a feedback endpoint, eval dashboard, LLM-as-judge grading mode to the evaluation script, and initiated a fine-tuning pipeline, concluding with a successful LoRA training smoke test on a substitute model.

Real-time token streaming was implemented, resolving a 502 error caused by Ollama's SSE format mismatching .json() parsing. A pooled HTTP client was deployed alongside measured baseline latency metrics (p50: 26.3s, p90: 88.7s from logged traffic). Two broken MCP servers (fetch, secedgar) incompatible with the installed SDK were silently disabled rather than left in a failed state. Executable setup scripts (Setup-MCP-Fetch.command, Setup-MCP-WebSearch.command) were built for one-click tool expansion. SOP-LLM-OPERATIONS.md was authored, formalizing fragmented tribal knowledge into a standardized daily runbook.

A 14.4-second delay on a trivial query was traced end-to-end through logs to its root cause: the model was needlessly filesystem-searching for non-existent documents rather than recognizing answers already in the system prompt. A layered fix was applied: initial prompt clarification, followed by a fast regex backstop, and finally a calibrated embedding-similarity threshold (0.80 based on logged scores). The diagnostic process and architectural reasoning were fully documented in AGENT_LATENCY_INVESTIGATION_SUMMARY.md, alongside this study-note set.

• Reactive over speculative: Every major architectural decision (memory scoping, retrieval ranking, latency gating, validity checks) emerged from debugging real usage, not theoretical planning.
• Local-by-design: The entire stack runs on bare Mac Studio hardware with zero managed cloud dependencies, isolated to a dedicated 1.8 TB APFS volume for maximum portability and backup simplicity.
• Documentation-first workflow: Drafting structural guides (KNOWLEDGE_MANAGEMENT_GUIDE.md, SYSTEM_OVERVIEW_SUMMARY.md) before implementation established the discipline needed to track rapid iteration.
• Sandbox boundaries dictate workflow: The AI's inability to make outbound requests (403 blocked-by-allowlist) necessitated manual script execution and shaped the entire knowledge-ingestion pipeline.
• Iterative stabilization: From initial baseline gaps (65% pass rate) to streaming fixes, MCP SDK compatibility checks, and embedding threshold calibration, continuous operational tuning transformed a prototype into a reliable system.