06 - Improvement Roadmap

The five-minute fixes that eliminate known risks and maintenance debt:

1. Move the Tavily API key out of mcp_servers.json Move it into an environment variable or a chmod 600 secrets file that the main config merely references. This is a five-minute change with a real, concrete downside currently sitting in plaintext: the key appears anywhere a user could commit the config file, back it up in cleartext, or expose it to unintended processes.

Done looks like: The key never appears in any file that could plausibly be committed, backed up in cleartext, or read by anything other than the process that needs it.

2. Copy a backup off the AI_DATA volume The backup system itself is solid; the gap is purely "off-machine." One manual copy to a different physical drive or cloud storage closes the single point of failure that would otherwise mean a drive failure loses everything, not just accidental deletions.

Set a recurring quarterly reminder, per the project's own SOP.

3. Apply the regex-then-embedding pattern to skills/loader.py The exact fix already exists in the codebase (Steps 5.9 / 5.10) for a structurally identical problem. This is closer to "port a working solution" than "solve a new problem" — a good first extension exercise that directly fixes a confirmed, logged false-positive.

4. Clean up orphaned files (memory_test2.db, stale wiki-ingest-state) Confirm they're safe to delete before removing them. Trivial, but it removes ambiguity for the next person (including a future you) reading the folder.

The higher-effort items that compound in value and are worth tackling while momentum is high:

5. Broaden the eval/test-question set beyond 20 questions 65% (13/20) on a 20-question baseline is a real signal — but a small one. A handful of new failing questions could swing the pass rate significantly without reflecting an actual regression; a handful of newly-passing questions could just as easily mask a real one.

Growing this to 50–100 questions, sourced from actual fumbles logged in production traffic (the project already captures feedback via logs/agents/feedback.jsonl), would make "did this change help or hurt?" a much more trustworthy question to answer.

6. Add a schema-validated request model to the main endpoint Swapping the current raw json.loads() for a proper request schema (Pydantic, since FastAPI already supports it natively) turns confusing internal errors into clean, actionable 400 responses for any malformed caller request — cheap to add, and it pays off every time an integration is being debugged.

7. Add a lightweight automated test run The project already has real end-to-end and streaming tests (test_endpoint_e2e.py, test_streaming.py) — the gap is that nothing runs them automatically. Wiring these into even a simple pre-restart check (before Restart-Agent-Server.command actually swaps in new code) would catch a regression before it reaches production traffic, closing the loop the project's own SOP already gestures at ("run the test suites" is step 3 of "updating server code," but nothing enforces it).

8. Build a human-approval gate for writes This is the one guardrail explicitly left unbuilt in the project's own guardrails checklist. Given the current write scope is already narrow (confined to the knowledge-bank folder), this doesn't need to be elaborate — even a simple pause and log, require a second explicit confirmation call pattern would close the gap between "sandboxed" and "supervised."

9. Manually fetch the remaining .mil / .gov-blocked documents Roughly two-dozen files across the GAO/service financial-report and DFAS/service-FM banks are known-missing because automated downloads are blocked, and the exact URLs are already documented in the setup guide's manual-download tables. This is pure execution, not engineering — the hard part (figuring out what's missing and where to get it) is already done.

10. Verify the remaining integrated projects individually Only one of roughly nineteen other projects pointed at this server has been directly confirmed to work correctly against the new agent-server; the rest are assumed to follow the same integration pattern. A short verification pass (confirm each project's environment variables include :8443 and a valid agent-server key, not a stale :443/gateway key) would catch silent breakage before a user does.

Optional improvements worth revisiting only when a specific, documented need triggers them:

11. Evaluate the fine-tuned adapter against the quality baseline The fine-tuning pipeline works end to end (proven by the LoRA smoke test), but training on ~13 examples was never expected to produce something worth deploying. Worth revisiting once:

• The eval suite (item 5) has generated meaningful more graded PASS examples to train on; and
• The underlying question of which production model to actually target is resolved (the smoke test deliberately used a small substitute model instead).

12. Investigate prompt-prefix caching alternatives if latency becomes a real complaint Ollama doesn't expose this control, and vLLM (the usual fix) doesn't run on Apple Silicon. If this ever becomes a genuine bottleneck rather than a documented limitation, the realistic paths are:

• A serving layer that supports prefix caching on Apple Silicon (worth researching directly — this space moves fast); or
• Reducing the size of the stable system-prompt prefix so there's simply less to reprocess each time.

13. Add HTTP/SSE MCP transport support if a needed tool only offers it Currently stdio-only by design, which has covered every tool wired in so far. Build this only when a specific, wanted MCP server turns out to require it — not speculatively (real new surface area to maintain).

14. Revisit horizontal scaling only if concurrent load genuinely requires more than one Mac Studio can serve The in-memory rate limiter, caches, and MCP session pool all assume a single process. Multi-instance deployment would need these moved to shared storage (Redis, or even just a shared SQLite file with appropriate locking) — real work, and not worth doing ahead of an actual capacity need since the current single-instance design is simpler and has no known bugs (only a known scaling ceiling).

15. Consider Tier 3 (graph) knowledge retrieval only if the broadened eval set (item 5) shows a real, repeated pattern of multi-hop relationship questions failing. The project's own knowledge-management guide states this correctly: don't build graph RAG speculatively. It's expensive to index and maintain, and vector RAG plus the curated wiki are explicitly expected to cover the large majority of real questions.

Items 1–4 are each under an hour of work and meaningfully reduce risk or maintenance debt — do these regardless of what else is prioritized. Items 5–10 compound: a bigger eval set (#5) makes every later change (including #11) easier to evaluate honestly, and closing the manual-download gaps (#9) plus verifying integrations (#10) are pure "finish what's already 90% done" work.

Items 11–15 are genuinely optional and should be triggered by an observed need, not built just because they're on a list — that discipline (build in response to real evidence, not in anticipation of it) is the same one that produced most of the fixes recorded in 00_Project_Timeline.md.

• Do three things this sprint: extract the Tavily key, copy a backup off-machine, and apply the existing regex-then-embedding pattern to skills/loader.py — hours of work for outsized risk reduction.
• The eval set is too small: 50–100 questions from production traffic would turn "did this change help or hurt?" into an honest question.
• Close two operational gaps manually: the blocked .mil/.gov documents and the unverified integration projects — pure execution, no engineering.
• Items 11–15 are optional: only pursue them when triggered by a specific, observed need rather than doing them because they're on a list.