03 - The Agent Harness, Deep Dive

This document details the architecture, request lifecycle, and core subsystems of apps/agent-server, a lean, hand-rolled agent harness built on FastAPI, httpx, ChromaDB, and the MCP SDK. No external frameworks are used; every layer—from authentication routing to memory management—is explicitly implemented to ensure predictable debugging and straightforward extension.

A request enters via POST /v1/chat/completions with a Bearer API key, formatted to match OpenAI's native API contract. Processing follows a strict sequence:

1. Authentication runs before the request body is ever parsed. auth.authenticate() validates that the bearer token's hash matches a known key in keystore.py (which stores only SHA-256 hashes; raw keys are displayed once at creation and never recoverable), confirms the key is enabled and unexpired, verifies required scopes (chat, tools, or admin), checks per-minute rate limits, and validates daily token quotas.
2. System prompt composition regenerates from scratch on every request. compose_system_prompt() concatenates:
   • The base identity, tool-use policy, and guardrails from AGENT.md (read directly from disk each time, allowing instant behavioral updates without restarts).
   • At most one matched skill block (keyword-driven via skills/loader.py).
   • A plain-language memory briefing for the specific user (_compose_memory_section(), pulling data from memory_store.py).
3. Self-referential routing intercepts the prompt before tool access is granted. is_self_referential() determines whether the query targets the agent itself rather than domain content. If true, tool access is stripped for that request only (detailed below).
4. Orchestration executes in graph.py, a lightweight custom engine that runs "waves" of work. All independent operations in a given step execute concurrently via asyncio.gather. Results merge into a shared state object, and the next wave is assembled from each node's declared follow-ups. Tool calls issued by the model spawn individual nodes in the subsequent wave, ensuring truly parallel execution rather than sequential polling.
5. Tool hooks enforce lifecycle boundaries. A pre-tool hook can block execution entirely (e.g., enforcing write-only boundaries within the knowledge-bank folder). A post-tool hook triggers side effects (e.g., initiating background re-indexing after successful writes). Both are fault-tolerant: a hook error logs and skips rather than crashing the request.
6. Model forwarding is abstracted into a single function. _prepare_upstream_body() handles all formatting for the Ollama API call. This is the definitive location to modify model invocation logic (see 07_Switching_LLM_Models_Guide.md).
7. Memory updates run asynchronously. Once the graph settles, memory_pipeline.analyze_and_record() fires as a background task. It never blocks or delays the response the caller receives.

Streaming requests follow a parallel but duplicated path. Because streaming requires continuous token emission, it bypasses graph.py's wave-based final-state model. Instead, run_streaming() hand-rolls the same planner-then-tools loop directly, reusing the exact same hooks and tool registry to maintain behavioral parity. Note: This duplication means tool-round budgets and hook ordering are implemented twice and require manual synchronization—a documented maintenance overhead addressed later.

tools/registry.py maintains two parallel data structures: a list of tool schemas (exposed to the model) and a dictionary mapping names to executable Python functions. Adding native tools like search_knowledge_base or get_study_plan requires implementing the function and registering it in both structures.

External tools integrate via the Model Context Protocol (MCP). mcp_client.py:

• Parses mcp_servers.json.
• Launches configured servers as local subprocesses.
• Queries each for available tools and registers them into the unified registry under a mcp__<server>__<tool> prefix.

This abstraction ensures agent_loop.py treats hand-written and external tools indistinguishably. Failure isolation is built in: if an MCP server fails to connect, it registers zero tools and emits a warning without disrupting core functionality. Currently, two configured servers (fetch, secedgar) remain disabled due to SDK incompatibility—a live demonstration of this failure boundary working as intended.

Active integrations:

• filesystem (scoped to the knowledge-bank directory)
• sequentialthinking (structured planning/reasoning)
• pdfreader
• tavily (web search)

Current constraint: Only the stdio transport is supported. MCP servers requiring HTTP or SSE cannot yet be connected through this client.

memory_store.py manages per-user facts (grade level, subject focus, documented knowledge gaps) in a partitioned SQLite database. Users are identified by caller-supplied identifiers, defaulting to the API key's ID if none is provided.

A critical architectural rule governs all writes:

Only explicit user input can update memory. Retrieved documents and model-generated text cannot write to or alter stored facts. This prevents hallucinated or retrieved claims from quietly becoming trusted state about the user.

Facts undergo a multi-day corroboration process before being promoted to stable status (trusted enough for confident assertion). Grade level specifically requires an explicit first-person statement from the user and can never be inferred from context.

Document retrieval in tools/knowledge_base.py follows a strict two-stage pipeline:

1. Embedding & Search: Queries are embedded using nomic-embed-text (the same model used for indexing) and matched against a ChromaDB collection via cosine similarity. This metric was deliberately chosen over the default distance metric because it measures vector direction rather than raw magnitude, which aligns better with dense text embeddings.
2. Source-Authority Re-ranking: A lightweight configurability layer nudge results toward higher-trust sources without overriding genuine relevance. This feature was added to resolve a real-world failure mode where secondary summary documents consistently outranked the primary regulations they were meant to support.

The routing gate acts as a low-cost filter before expensive reasoning begins:

1. Regex Tier: A curated list checks for exact phrasings (e.g., "who are you," "what guardrails do you follow"). It strictly caps matching to short questions and excludes queries containing domain vocabulary (preventing false positives like budget adjustment questions that inadvertently use shared words).
2. Embedding Tier: Triggered only if regex misses the query. The prompt is embedded, compared against canonical "about the agent" examples via cosine similarity, and evaluated against a 0.80 threshold. This value was empirically calibrated from logged similarity scores to maximize safe margin against false positives while catching genuine self-referential prompts.

Every routing decision (including near-misses) logs to logs/agents/triage.jsonl, enabling future threshold tuning against production data rather than relying on static configuration.

Code-enforced guardrails include:

• Write-path hooks that fail closed, blocking unrecognized or malformed write operations by default.
• Mandatory logging for every tool call.
• Strict credential isolation: the agent loop is never passed external credentials (e.g., Tailscale admin keys, backup drive access) and operates solely within its sandboxed folder.

Documented gap: No human-approval pause exists before a write executes. The folder sandbox remains the sole barrier against immediate execution of mistaken or manipulated outputs.

The harness is built for predictability and extensibility, but carries intentional trade-offs:

Both trade-offs are fully cataloged with resolution paths in 05_Gap_Analysis.md and 08_Extending_The_Harness_Agent.md.

• Request processing is strictly sequential and transparent: Auth → prompt composition → routing check → wave-based orchestration → model forwarding → background memory update.
• Tools are framework-agnostic: A unified registry abstracts hand-written functions and MCP-integrated tools into identical callable schemas, with graceful failure isolation for external servers.
• Memory is strictly user-driven: ChromaDB retrieval and model generation never alter stored state; facts require multi-day corroboration to become stable.
• Guardrails are code-enforced, not aspirational: Writes fail closed, credentials never leave the sandbox, and tool calls are fully audited. Scaling and streaming path duplication remain the primary areas requiring architectural evolution.