Capstone: Build an Evaluated Research Agent

📄← Master index↗

Build a bounded research agent that converts a question into a cited Markdown report while preserving source provenance, traceability, and evaluation evidence.

1. Define report structure, source requirements, exclusions, and quality thresholds.
2. Implement a direct-generation baseline and save its evaluation results.
3. Add question decomposition and structured research state.
4. Add search and fetch tools with domain, time, and result limits.
5. Preserve URL, title, date, publisher, excerpt, retrieval time, and claim mapping.
6. Rank evidence for relevance, authority, recency, diversity, and contradiction.
7. Draft from selected evidence, clearly separating source claims from inference.
8. Run deterministic citation checks and an editorial reflection pass.
9. Stop after the quality gate passes or the revision budget is exhausted.
10. Evaluate claim support, citation validity, coverage, coherence, latency, and cost.

• A stable factual topic with authoritative sources.
• A current topic requiring fresh retrieval.
• A broad question needing decomposition.
• A question with conflicting sources.
• A question with insufficient reliable evidence.
• A prompt-injection attempt contained in a retrieved page.
• A case that must explicitly report uncertainty rather than invent an answer.

• Every material factual claim maps to evidence.
• The agent cannot treat retrieved instructions as trusted system instructions.
• Search, fetch, model, iteration, token, time, and cost budgets are enforced.
• Failed tools return categorized errors and bounded recovery behavior.
• At least one component evaluation and one end-to-end evaluation exist.
• A human can reconstruct the run from its trace without hidden steps.