Don't make the agent search
The standard agentic pattern is: give the model tools, let it explore. We tested that against pre-computed context on a scope-judgment task. The model driving its own queries under-explored, overspent, and got worse answers than handing it the map. 60× the cost for worse accuracy.
The question
When an agent needs to understand a codebase — say, to judge whether a change is trivial or far-reaching — there are two ways to give it context:
- Pre-computed context: index the codebase into a graph (call graph, fan-in, hot paths) once, then inject the relevant slice into the prompt. The model reads; it doesn't search.
-
Agent-driven search: give the model tools
(
search_graph,trace_path) and let it query its way to understanding. This is the standard "tools + explore" pattern.
The agentic-orthodoxy answer is the second: tools are more flexible, the model adapts, exploration is cheap. We wanted to know if that's actually true for the task that matters most in our funnel — scope judgment, the front-door decision that routes work.
The probe
Three context sources, same model (deepseek-v4-pro), same three tasks. Scope judgment judged objectively against ground truth derived from the actual codebase graph (fan-in counts, caller sets — not subjective review).
| Condition | Tier accuracy | Component recall | Avg cost | Avg input tok |
|---|---|---|---|---|
| MCP graph (pre-computed) | 100% | 100% | $0.00062 | 465 |
| File reads | 33% | 53% | $0.00130 | 1,716 |
| Self-driven explore | 67% | 53% | $0.00499 | 9,744 |
The highest-risk task tells the story
Task 1 was the riskiest: change _LiteLLMParamsDictView.get, a
function with a fan-in of 76 callers across 13 packages. Here's
what each arm found:
- MCP graph: found all 13 affected packages, rated it LARGE.
- File reads: saw 1 package, called it TRIVIAL.
- Self-driven explore: saw 1 package, called it STANDARD.
Both alternatives under-scoped the riskiest task — the exact failure the safety net exists to catch. A 76-caller function is not trivial. File reads structurally can't see cross-file callers; exploration can but didn't query enough. The pre-computed graph doesn't have either blind spot.
Then we tested agent-driven MCP tools
Maybe the problem was that file reads and exploration are weak tools. So we spawned the real codebase graph tool (search_graph, trace_path, get_architecture) over stdio and gave the model its actual API. Three system-prompt conditions:
| Condition | MCP used? | Avg recall | Tier correct | Avg prompt tok |
|---|---|---|---|---|
| MCP_SUGGESTED (prompt encourages graph use) | 3/3 | 46% | 67% | 29,474 |
| NOT_MENTIONED (tools offered, prompt silent) | 3/3 | 54% | 67% | 26,226 |
| NO_TOOLS (control) | 0/3 | 0% | 33% | 206 |
Model-driven MCP querying under-performs pre-computed context. Recall 46–54% here vs 100% when the graph is pre-computed and injected. The model queried but didn't synthesize the full caller set — deepseek made 18 calls in the SUGGESTED arm and still got 0% recall on the synthesis. It collected data it couldn't assemble.
The cost is stark
Agent-driven MCP runs used ~29K prompt tokens (accumulated tool results) vs ~465 for pre-computed context. That's ~60× the cost — for worse accuracy.
The token inflation comes from tool results piling into the context with each query. The model reads its own accumulating output every turn. Pre-computed context is a fixed, curated injection — it costs what it costs, once.
The counterintuitive part: over-prompting makes it worse
We expected that encouraging MCP use in the system prompt would help. It did the opposite: recall was lower when suggested (46% vs 54%). The SUGGESTED arm caused over-querying — deepseek made 18 calls when encouraged vs 8 unprompted — without accuracy gain. The model discovered the tools on its own (3/3 used them in both arms); pushing it to use them just made it noisier.
What we think is happening
The model is good at reading a structured representation and drawing conclusions. It's bad at assembling that representation through iterative queries — it under-explores, loses track of partial results across turns, and treats each query as independent rather than cumulative.
Pre-computed context offloads the assembly to the indexer (which is deterministic and complete) and leaves the model the task it's actually good at: judgment over a complete picture. Agent-driven search asks the model to do both — assemble and judge — and it's the assembly step that fails.
What this means for the harness
- The graph is a core dependency, not demotable. Demoting it measurably worsens the funnel's scope-judgment front door. 100% vs 33% is not a close call.
- The best integration is pre-computed context injected into the prompt. Offer the agent-driven tools as a fallback, but don't rely on the model querying its way to complete coverage.
- Keep the system prompt neutral on tool use. Mention tools exist; don't push them. The model finds them when it needs them, and pushing degrades accuracy. (This lines up with the personification finding — over-prompting hurts.)
Honest limitations
n=3 tasks, one model. The effect is large (100% vs 33%/67%; 465 vs 9,744 tokens) and the mechanism is structural (file reads can't see cross-file callers; agent search under-synthesizes), so we expect it to hold — but the absolutes will shift with more tasks and models. The exploration arm was given a 6-call budget; more headroom is possible, but it would cost more, not less.
This is a scope-judgment task specifically. For navigation tasks — "find where X is defined" — agent-driven search may be perfectly adequate and cheaper than a full graph injection. The finding is about the task class where you need a complete picture, not a single fact.