Local Server Latency Investigation

What started as "why did one answer take 14 seconds" turned into a real diagnosis of apps/agent-server's request pipeline, two shipped fixes, and a calibration pass backed by real logged data. This document walks through each step in order, what was found, what was changed, and why — so the reasoning survives even if the specific numbers get retuned later.

A simple question ("what guardrails apply") took 14.4s to first token, 16.2s total, with an unusually large input (31,664 tokens) for such a short question. That size mismatch — trivial question, huge prompt — was the first clue something structural was wrong, not just "the model is slow."

Why this mattered: latency complaints are easy to misdiagnose as "the hardware needs to be faster" or "the model is too big." Chasing that without evidence would have led to the wrong fix (buying more compute, switching models) instead of the actual, free fix (stop doing unnecessary work).

Explored logs/agents/*.jsonl (requests.jsonl, tool_calls.jsonl, skill_matches.jsonl) and logs/ollama.log, and read the serving code (apps/agent-server/agent_loop.py, main.py, graph.py).

Why: this system logs every request, every tool call, and every skill match with timestamps and token counts, specifically so a slow request is reconstructable after the fact. Guessing at causes when the evidence is sitting in a log file wastes time and risks fixing the wrong thing.

The slow request's tool_calls.jsonl entry showed the model made four sequential filesystem searches — for a file literally named AGENTS.md, for anything matching *guardrail*, a directory listing, then agent*/*guide* — all before answering the question from its own system prompt on the fifth attempt. Cross-referencing ollama.log confirmed five separate sequential model calls (4.84s + 2.99s + 1.39s + 2.30s + 4.12s ≈ 15.6s), matching the total latency almost exactly.

Why this happened: AGENT.md (the base system prompt) told the model to "prefer search over answering from memory for factual, regulatory questions," with no exception for questions about the agent's own identity or rules. A question about guardrails read as "factual," so the model went looking for a document instead of noticing the answer was already three paragraphs below in the same prompt it was reading. On top of that, it reached for raw filename-guessing tools instead of the semantic search tool it was told to prefer — a second, compounding mistake.

Before touching anything, assessed the existing engine (graph.py's wave-based executor) against how systems like Claude Code are built. Conclusion: the execution engine was already solid — genuine concurrent tool-call fan-out within a wave, prompt-prefix caching, streaming, pooled connections. The gap was entirely in instruction, not machinery: the model was never told to batch independent searches, disambiguate which tool to use for which job, or recognize questions about itself as a special case.

Why start here instead of jumping to code: a wrong architectural diagnosis (e.g. "the engine can't parallelize") would have led to rebuilding something that already worked, instead of fixing the actual gap (what the model is told to do with what it already has).

Added three things to the system prompt, kept everything else (citation requirements, no-fabrication rule, tool-scope honesty) untouched:

1. A narrow "answering questions about yourself" carve-out — with a worked example distinguishing "what guardrails apply to you" (answer directly) from "what are the DoD FMR's internal-control guardrails" (still a real search, even though both use the word "guardrails").
2. Tool disambiguation — use semantic search_knowledge_base for content questions; only use filesystem tools when the path is already known, never to guess at a filename.
3. A parallel-batching instruction — issue independent tool calls together in one turn instead of one at a time, since the engine already supports concurrent execution within a wave; it just wasn't being used.

Why a narrow carve-out, not a blanket "never search" rule: the cost of an unnecessary search is a few seconds; the cost of wrongly refusing a real lookup is a wrong or unsupported answer. Every edit was written to fail toward more searching when genuinely unsure, never less.

A prompt instruction is a request the model can still misjudge. Added is_self_referential() in agent_loop.py — a hard, code-level gate that runs before the model ever sees the request: if the question matches a narrow, curated set of self-referential phrasings ("who are you," "what are your instructions," etc.), tools are switched off for that one request, guaranteeing zero wasted search rounds regardless of what the model would have decided.

Why biased toward precision over recall: a false negative (missing a self-referential question) just falls through to the Step 5 prompt fix — no regression, same as before. A false positive (wrongly blocking a real domain lookup) would actively break a legitimate answer, which is strictly worse than the latency this exists to fix. So the gate only fires on high-confidence literal matches, with a word-count cap and a domain-vocabulary exclusion list as extra safety nets.

You asked the exact same question, reworded ("what is the guardrail apply before agent can act" instead of "what guardrails apply"). The regex gate missed it — no exact phrase match — but the Step 5 prompt fix still worked; the model didn't search. Token count dropped from 31,664 to 6,710 confirming zero wasted tool rounds. But total latency was still 13.6s, and investigation showed why: a cold model load (5.6s to reload the 35B model into memory) because 2h10m had passed since the last request — well past the 30-minute keep-alive window. Unrelated to the restart; Ollama's own idle timer, working as designed.

Why this mattered: it proved the regex gate, while safe, doesn't generalize to paraphrasing — which is exactly the kind of gap a rigid rule-based system has and a model-driven one doesn't.

Rather than trying to enumerate every possible paraphrase by hand, added a second tier: if regex misses (but the question still passes the same precision guards), embed the question using the same nomic-embed-text model already powering search_knowledge_base, and compare it by cosine similarity against a small set of canonical "about-the-agent" example questions. Above a threshold → treat as self-referential, same as a regex hit.

Why embeddings instead of more regex patterns: regex can only catch phrasings someone thought to write down. Nearest-neighbor similarity in embedding space generalizes to paraphrases no one anticipated, at a fraction of the cost of a full model call (one embedding lookup, no token generation — typically well under a second vs. a multi-second generation round trip). This mirrors how production agent systems layer cheap, narrow logic in front of expensive, flexible reasoning: deterministic first, semantic second, full model only when neither resolves confidently.

Built with a fail-safe: any embedding-call failure (model not pulled, network hiccup) falls through to the existing path rather than breaking the request — this is a latency optimization, not a safety feature, so it must never be the reason a request fails.

The threshold (started at 0.84, a guess — this sandbox has no network path to your Mac's Ollama instance, so it couldn't be tuned from here) needed real calibration. Built two rounds of test questions and ran them through the live server:

• Round 1 confirmed the target case now works (0.8846, above threshold) and surfaced two "near miss" paraphrases (0.8221, 0.6978) that didn't clear the bar.
• Round 2 deliberately added domain questions with no obvious marker words (so they'd actually reach the embedding tier instead of being excluded beforehand) — including one intentionally phrased to mirror the agent-facing structure ("what rules apply before you can close an office") — plus one deliberately ambiguous case ("what rules do I need to follow," about the user, not the agent).

Why this two-round design: the first round proved the fix works on the happy path. The second round tried to break it — the only way to find the real boundary between "catches paraphrases" and "wrongly blocks real questions" is to throw adversarial, boundary-hugging cases at it and read the actual scores, not assume the boundary is wherever felt safe.

Real scores from triage.jsonl: true self-referential questions clustered 0.70–0.88. The closest adjacent non-match — the ambiguous "about the user" case — scored 0.7245. Every real domain question, including the adversarial one, scored below 0.69.

Set the threshold to 0.80: catches one more real paraphrase (0.8221) with a 0.075 margin above the nearest false-positive risk. Deliberately did not lower it further to catch a 0.7466 case — that would leave only a 0.02 margin above the 0.7245 ambiguous case, too thin to trust without more data specifically testing that boundary.

Why stop at 0.80 instead of chasing every miss: every threshold move trades recall (catching more real cases) against precision (risking a false positive that breaks a real answer). Given the stated priority — false positives are strictly worse than false negatives — the right stopping point is "the last move with a comfortable, evidence-backed margin," not "the lowest number that still technically works today."

Layer

What it does

Cost when it doesn't apply

AGENT.md prompt fix

Tells the model directly not to search for self-referential questions, disambiguates tools, encourages batching

None — it's just better instructions

Regex gate (Step 5.9)

Hard-blocks tools on exact-phrase self-referential matches

Zero — pure Python, no model call

Embedding gate (Step 5.10)

Catches paraphrases regex misses, via semantic similarity

One embedding call (~sub-second), only when regex misses and guards pass

Threshold = 0.80

Decision boundary for the embedding gate

Calibrated against real logged scores, not assumed

Still open / worth doing next, in rough priority order:

1. Keep an eye on logs/agents/triage.jsonl as real traffic accumulates — the embedding_near_miss entries are free calibration data for deciding whether 0.80 needs to move.
2. The DFAS "segregation of duties" test question took 5 search rounds and 56 seconds — a separate, legitimate-retrieval latency story (not a routing bug) that hasn't been investigated yet.
3. skills/loader.py's keyword-overlap skill matcher has known false positives (e.g., transformer-architecture notes matching the K12 skill) — same class of problem as Steps 5–8 solved for self-referential routing, not yet applied there.— Summary (2026-07-30)

What started as "why did one answer take 14 seconds" turned into a real diagnosis of apps/agent-server's request pipeline, two shipped fixes, and a calibration pass backed by real logged data. This document walks through each step in order, what was found, what was changed, and why — so the reasoning survives even if the specific numbers get retuned later.

A simple question ("what guardrails apply") took 14.4s to first token, 16.2s total, with an unusually large input (31,664 tokens) for such a short question. That size mismatch — trivial question, huge prompt — was the first clue something structural was wrong, not just "the model is slow."

Why this mattered: latency complaints are easy to misdiagnose as "the hardware needs to be faster" or "the model is too big." Chasing that without evidence would have led to the wrong fix (buying more compute, switching models) instead of the actual, free fix (stop doing unnecessary work).

Explored logs/agents/*.jsonl (requests.jsonl, tool_calls.jsonl, skill_matches.jsonl) and logs/ollama.log, and read the serving code (apps/agent-server/agent_loop.py, main.py, graph.py).

Why: this system logs every request, every tool call, and every skill match with timestamps and token counts, specifically so a slow request is reconstructable after the fact. Guessing at causes when the evidence is sitting in a log file wastes time and risks fixing the wrong thing.

The slow request's tool_calls.jsonl entry showed the model made four sequential filesystem searches — for a file literally named AGENTS.md, for anything matching *guardrail*, a directory listing, then agent*/*guide* — all before answering the question from its own system prompt on the fifth attempt. Cross-referencing ollama.log confirmed five separate sequential model calls (4.84s + 2.99s + 1.39s + 2.30s + 4.12s ≈ 15.6s), matching the total latency almost exactly.

Why this happened: AGENT.md (the base system prompt) told the model to "prefer search over answering from memory for factual, regulatory questions," with no exception for questions about the agent's own identity or rules. A question about guardrails read as "factual," so the model went looking for a document instead of noticing the answer was already three paragraphs below in the same prompt it was reading. On top of that, it reached for raw filename-guessing tools instead of the semantic search tool it was told to prefer — a second, compounding mistake.

Before touching anything, assessed the existing engine (graph.py's wave-based executor) against how systems like Claude Code are built. Conclusion: the execution engine was already solid — genuine concurrent tool-call fan-out within a wave, prompt-prefix caching, streaming, pooled connections. The gap was entirely in instruction, not machinery: the model was never told to batch independent searches, disambiguate which tool to use for which job, or recognize questions about itself as a special case.

Why start here instead of jumping to code: a wrong architectural diagnosis (e.g. "the engine can't parallelize") would have led to rebuilding something that already worked, instead of fixing the actual gap (what the model is told to do with what it already has).

Added three things to the system prompt, kept everything else (citation requirements, no-fabrication rule, tool-scope honesty) untouched:

1. A narrow "answering questions about yourself" carve-out — with a worked example distinguishing "what guardrails apply to you" (answer directly) from "what are the DoD FMR's internal-control guardrails" (still a real search, even though both use the word "guardrails").
2. Tool disambiguation — use semantic search_knowledge_base for content questions; only use filesystem tools when the path is already known, never to guess at a filename.
3. A parallel-batching instruction — issue independent tool calls together in one turn instead of one at a time, since the engine already supports concurrent execution within a wave; it just wasn't being used.

Why a narrow carve-out, not a blanket "never search" rule: the cost of an unnecessary search is a few seconds; the cost of wrongly refusing a real lookup is a wrong or unsupported answer. Every edit was written to fail toward more searching when genuinely unsure, never less.

A prompt instruction is a request the model can still misjudge. Added is_self_referential() in agent_loop.py — a hard, code-level gate that runs before the model ever sees the request: if the question matches a narrow, curated set of self-referential phrasings ("who are you," "what are your instructions," etc.), tools are switched off for that one request, guaranteeing zero wasted search rounds regardless of what the model would have decided.

Why biased toward precision over recall: a false negative (missing a self-referential question) just falls through to the Step 5 prompt fix — no regression, same as before. A false positive (wrongly blocking a real domain lookup) would actively break a legitimate answer, which is strictly worse than the latency this exists to fix. So the gate only fires on high-confidence literal matches, with a word-count cap and a domain-vocabulary exclusion list as extra safety nets.

You asked the exact same question, reworded ("what is the guardrail apply before agent can act" instead of "what guardrails apply"). The regex gate missed it — no exact phrase match — but the Step 5 prompt fix still worked; the model didn't search. Token count dropped from 31,664 to 6,710 confirming zero wasted tool rounds. But total latency was still 13.6s, and investigation showed why: a cold model load (5.6s to reload the 35B model into memory) because 2h10m had passed since the last request — well past the 30-minute keep-alive window. Unrelated to the restart; Ollama's own idle timer, working as designed.

Why this mattered: it proved the regex gate, while safe, doesn't generalize to paraphrasing — which is exactly the kind of gap a rigid rule-based system has and a model-driven one doesn't.

Rather than trying to enumerate every possible paraphrase by hand, added a second tier: if regex misses (but the question still passes the same precision guards), embed the question using the same nomic-embed-text model already powering search_knowledge_base, and compare it by cosine similarity against a small set of canonical "about-the-agent" example questions. Above a threshold → treat as self-referential, same as a regex hit.

Why embeddings instead of more regex patterns: regex can only catch phrasings someone thought to write down. Nearest-neighbor similarity in embedding space generalizes to paraphrases no one anticipated, at a fraction of the cost of a full model call (one embedding lookup, no token generation — typically well under a second vs. a multi-second generation round trip). This mirrors how production agent systems layer cheap, narrow logic in front of expensive, flexible reasoning: deterministic first, semantic second, full model only when neither resolves confidently.

Built with a fail-safe: any embedding-call failure (model not pulled, network hiccup) falls through to the existing path rather than breaking the request — this is a latency optimization, not a safety feature, so it must never be the reason a request fails.

The threshold (started at 0.84, a guess — this sandbox has no network path to your Mac's Ollama instance, so it couldn't be tuned from here) needed real calibration. Built two rounds of test questions and ran them through the live server:

• Round 1 confirmed the target case now works (0.8846, above threshold) and surfaced two "near miss" paraphrases (0.8221, 0.6978) that didn't clear the bar.
• Round 2 deliberately added domain questions with no obvious marker words (so they'd actually reach the embedding tier instead of being excluded beforehand) — including one intentionally phrased to mirror the agent-facing structure ("what rules apply before you can close an office") — plus one deliberately ambiguous case ("what rules do I need to follow," about the user, not the agent).

Why this two-round design: the first round proved the fix works on the happy path. The second round tried to break it — the only way to find the real boundary between "catches paraphrases" and "wrongly blocks real questions" is to throw adversarial, boundary-hugging cases at it and read the actual scores, not assume the boundary is wherever felt safe.

Real scores from triage.jsonl: true self-referential questions clustered 0.70–0.88. The closest adjacent non-match — the ambiguous "about the user" case — scored 0.7245. Every real domain question, including the adversarial one, scored below 0.69.

Set the threshold to 0.80: catches one more real paraphrase (0.8221) with a 0.075 margin above the nearest false-positive risk. Deliberately did not lower it further to catch a 0.7466 case — that would leave only a 0.02 margin above the 0.7245 ambiguous case, too thin to trust without more data specifically testing that boundary.

Why stop at 0.80 instead of chasing every miss: every threshold move trades recall (catching more real cases) against precision (risking a false positive that breaks a real answer). Given the stated priority — false positives are strictly worse than false negatives — the right stopping point is "the last move with a comfortable, evidence-backed margin," not "the lowest number that still technically works today."

Layer

What it does

Cost when it doesn't apply

AGENT.md prompt fix

Tells the model directly not to search for self-referential questions, disambiguates tools, encourages batching

None — it's just better instructions

Regex gate (Step 5.9)

Hard-blocks tools on exact-phrase self-referential matches

Zero — pure Python, no model call

Embedding gate (Step 5.10)

Catches paraphrases regex misses, via semantic similarity

One embedding call (~sub-second), only when regex misses and guards pass

Threshold = 0.80

Decision boundary for the embedding gate

Calibrated against real logged scores, not assumed

Still open / worth doing next, in rough priority order:

1. Keep an eye on logs/agents/triage.jsonl as real traffic accumulates — the embedding_near_miss entries are free calibration data for deciding whether 0.80 needs to move.
2. The DFAS "segregation of duties" test question took 5 search rounds and 56 seconds — a separate, legitimate-retrieval latency story (not a routing bug) that hasn't been investigated yet.
3. skills/loader.py's keyword-overlap skill matcher has known false positives (e.g., transformer-architecture notes matching the K12 skill) — same class of problem as Steps 5–8 solved for self-referential routing, not yet applied there.