Agent Design Doc · Working Demo
Open-Source Intelligence-to-Decision Agent
An architecture for turning open data into rigorous, mission-relevant products — at scale
A documented agent pattern drawn from two systems I built: a published open-source defense-industrial-base analysis, and HIGHGROUND, a strategy-advisor agent. This generalizes both into one reusable design — how the agent behaves, where it's allowed to act alone, and how I know its output is rigorous.
Katherine Hathaway · click any stage to expand
01 The pipeline
Intent in, decision-ready product out. Each stage names the tool it calls and whether a human checks it.
▸ tap a stage to see the detail + the real work it's drawn from
Frame the question intent
Translate a messy mission need into a structured analytic question: what decision does this serve, what would change the answer, what's the disconfirming evidence. Mirrors the Priority Intelligence Requirement discipline from military planning.
Real basis: framed the DIB study as a falsifiable question — "has the new defense-tech ecosystem actually diversified the industrial base, or replicated its concentration?"
Collect from open sources tool: API / retrieval
Pull structured open data via API; standardize and de-duplicate entities so the dataset is analysis-ready (schemas, naming, grain).
Real basis: built a Python pipeline against the USASpending.gov API (pandas, requests) with subsidiary-name standardization for the DIB paper.
Analyze at scale tool: compute + LLM
Run the scoring or model, then use the LLM to summarize, cluster, and surface what matters — operating across far more data than a human could read, while the analyst sets the logic.
Real basis: designed a multi-factor fragility score (HHI + funding concentration + single-firm dominance) across six capability clusters; LLM as analytical partner.
Check rigor eval gate
Before anything ships, the output runs the rigor rubric (Section 04). Low-confidence or unsourced claims get flagged and routed to a human — the agent does not present them as findings.
Real basis: applied an explicit analytic-rigor standard as the quality bar in the DIB paper, with a limitations section distinguishing robust findings from cautions.
Deliver mission-relevant product output
Package the finding in the format the decision-maker actually uses — a one-pager, a brief, a chart — with sourcing and confidence attached. Communication is part of the product, not an afterthought.
Real basis: published paper with four decision-ready charts; years of executive-brief and decision-support production.
02 Behavior primitives
The dials that define how the agent acts — set deliberately, not left to default.
obj Objective
Produce a rigorous, decision-relevant answer to the framed question — not the maximum volume of output. Optimizes for the decision, not the word count.
con Constraints
Open sources only; every claim carries a source and a confidence; no fabricated specificity. Stays inside the lane the question defines.
tool Tool calls
API retrieval, compute for scoring, LLM for synthesis. The agent chooses tools to serve the question and shows its work.
esc Escalation
When confidence drops or sources conflict, the agent stops and routes to a human rather than papering over the gap.
03 Trust tiers
Not every output earns the same autonomy. What the agent ships alone vs. what a human signs.
AUTO
Routine retrieval, standardization, and descriptive summaries with high source confidence — the agent runs and ships these on its own.
REVIEW
Analytical judgments, scores, and rankings — the agent produces them; a human checks the logic before they reach a decision-maker.
HUMAN
Conflicting sources, low confidence, or anything feeding a high-consequence call — the agent surfaces options; the human decides.
04 Rigor rubric
How I know the output is good. The eval gate in Stage 04 runs every line against this.
| Test | Question | Fail action |
|---|
| Sourced | Does every claim trace to an identifiable open source? | → flag unsourced claims |
| Reproducible | Could someone re-run the pipeline and get the same result? | → route to human |
| Calibrated | Is the confidence stated, and does it match the evidence? | → downgrade the claim |
| Falsifiable | Did we test the disconfirming case, not just the supporting one? | → add limitations note |
| Decision-fit | Does the format match how the decision-maker actually works? | → reformat product |
05 What this demonstrates
Each capability mapped to the role that asks for it.
Leveraging AI/ML tools to operate at scale on geopolitical / open-source analysisProduct Ops
Curating rigorous, mission-relevant products from raw dataProduct Ops
Improving operational processes & tradecraft (repeatable pipeline + eval loop)Product Ops
LLM-based systems & defining how an agent behaves (objectives, constraints, trust tiers)Mission Agents
Evaluation criteria — knowing when output is good enough to shipMission Agents
Rapidly prototyping with AI coding tools (this demo)Mission Agents
Built, not imagined. This architecture generalizes two real systems — a published open-source DIB analysis (Python pipeline + novel scoring framework) and HIGHGROUND, a strategy-advisor agent with a curated public-document knowledge base. The pattern is the through-line; both ends are work I've shipped.