05 - Gap Analysis

This document collects every known gap, limitation, and open issue currently on record across the project's own documentation and codebase — not speculation, but things the project itself has already flagged, plus a handful of observations from reading the actual code. Organized by category, roughly ordered by how much it should matter to you.

A live API key sits in plaintext in a config file. apps/agent-server/mcp_servers.json stores the Tavily web-search API key directly in the JSON the code reads — the file's own comment self-documents this. There's no secrets-manager indirection anywhere in the project. Anyone who can read that file (or a backup of it, or a git commit if it were ever accidentally added) has the key.

No human-approval gate before a write executes. The write-path guard restricts where the agent can write (only inside the knowledge-bank folder), but nothing pauses for confirmation before a write actually happens. This was explicitly called out in AGENT_SERVER_GUIDE_v2.md's own guardrails checklist as the one item left unbuilt.

Rate limiting and caches are single-process, in-memory. The per-key rate limiter, the AGENT.md cache, the embedding cache for the self-referential gate, and the MCP session pool are all plain in-process dictionaries. This is completely fine at today's scale (one process, one Mac) but would silently stop working correctly the day agent-server runs as more than one instance behind a load balancer — each instance would independently allow up to the configured limit.

Backups live on the volume they protect. Covers accidental deletion and bad edits; does not cover drive failure. The project's own SOP flags this and recommends copying a backup off-machine at least quarterly — as of the last recorded status, this had not yet become a routine habit.

Two knowledge banks have real, acknowledged coverage gaps, not because of a bug but because the source documents are genuinely hard to obtain automatically: roughly 10 of 13 GAO/service financial-report files and 14–16 of 17 DFAS/service-FM files are blocked by .mil/.gov sites that reject automated downloads and need to be fetched manually. K-12's 03-Lesson-Materials/ folder and DOD-FM's 03-SOPs-Internal/, 04-Reference-Data/, and 05-Examples/ folders are at 0% — the project's own documentation notes these categories can only ever be filled from the owner's own organizational content, not the public internet, and are likely to stay empty until someone deliberately adds that material.

The agent's own sandbox cannot make outbound network requests. Every knowledge-download script has to be run by the human owner in their own Terminal — this is a standing operational constraint, not a one-time bug, and it shapes how any future bulk-ingestion work has to happen.

The 20-question quality baseline currently sits at 65% (13 PASS / 7 FAIL). That's a real, measured number, not a guess — but it's also a small sample. Of the 7 failures at last measurement, most were traced to genuine retrieval-quality gaps (the answer exists in the source text but isn't being surfaced well), one was a ranking issue (fixed by source-authority weighting), and one was an answer-depth complaint (fixed by adjusting retrieval depth). Broader test coverage would give more confidence that fixes generalize rather than just patching the specific questions that were tested.

No cross-request prompt caching. A large, stable system-prompt prefix (roughly 4,400 tokens at the median) gets re-sent and reprocessed on every single request. Other serving stacks (and some cloud providers) cache a repeated prompt prefix to skip re-processing it — Ollama exposes no control for this, and the project's own SOP names this as a real, unresolved latency cost with no available fix short of switching serving layers entirely.

vLLM, the standard fix for FIFO request queuing, is not available on this hardware. It requires CUDA and does not run on Apple Silicon at all — so the parallel-request bottleneck has to be managed by tuning Ollama's own NUM_PARALLEL and context-length settings rather than solved outright.

Streaming and non-streaming requests are two separate implementations of the same control flow. graph.py's wave executor handles non-streaming requests; the streaming path hand-rolls an equivalent planner-then-tools loop because it needs to emit tokens as they arrive. Any future change to tool-round budgeting, hook execution order, or memory recording has to be applied in both places by hand — they're not sharing one code path.

The skills-matching system has the exact same false-positive risk that was just fixed elsewhere. skills/loader.py matches skills by keyword overlap, and a confirmed real case exists where content about a completely unrelated technical topic matched a K-12-specific skill purely on shared vocabulary. The two-tier regex-then-embedding fix built for the self-referential routing gate (Steps 5.9/5.10) was explicitly flagged as directly reusable here — but hasn't been applied yet.

The self-referential routing threshold is calibrated on a small, real-but-thin sample. 0.80 wasn't guessed, but it also wasn't tested against a large adversarial set — a new phrasing style or a genuine edge case could silently start disabling tool access for a legitimate domain question, and there's no automated regression test watching for that.

No schema validation on incoming request bodies. The main endpoint parses raw JSON rather than validating against a defined schema, so a malformed request tends to surface as an opaque internal error rather than a clean, helpful 400 response.

Two MCP servers are wired in but dead on arrival. fetch and secedgar both crash on import against the currently installed MCP SDK version — correctly disabled rather than left silently broken, but a real signal that the MCP Python ecosystem's API surface is still actively shifting, and any new server added in the future should be smoke-tested in Terminal before being trusted.

Memory's knowledge-gap detection is plain substring and regex matching. Cheap and easy to audit, but brittle — a student expressing confusion in an unexpected phrasing, or in a way the regex list didn't anticipate, won't be recorded, and nothing surfaces an error when that happens; it just silently doesn't capture the signal.

No automated test suite runs in CI. Real end-to-end and streaming tests exist (test_endpoint_e2e.py, test_streaming.py) and quality is measured via the graded test-question run, but nothing is wired to run these automatically on a change — correctness currently depends on a human remembering to run them.

The fine-tuning pipeline is a real but very early stub. Only 13 graded-PASS examples exist to train on — explicitly flagged in the project's own code as too few to produce a meaningful adapter. One real LoRA training run did complete successfully, but against a small substitute model, not the actual production model, because the production model's obvious MLX conversion turned out to be a vision-language variant incompatible with the plain text-tuning tool used. Evaluating a trained adapter against the quality baseline — the step that would actually prove this is worth continuing — hasn't happened yet.

One build step (5.8) is referenced in code but was never written up in the engineering guide — a small, acknowledged documentation gap rather than a functional one.

A few small orphaned files exist (memory_test2.db, a stale wiki-ingest state file, .webui_secret_key sitting loose) — flagged as likely-harmless dead weight, not yet cleaned up.

Only one of the ~19 other integrated projects has been directly verified to be correctly wired to the new agent-server; the rest are assumed to follow the same pattern but haven't been individually confirmed.

None of this is a crisis — the project's own build history shows a consistent pattern of finding exactly these kinds of gaps through real usage and fixing them with a narrow, well-reasoned mechanism. What's listed here is simply the current, honest snapshot: known, written-down, not yet done. 06_Improvement_Roadmap.md turns this into a prioritized plan.