How I accidentally benchmarked grep, three times

Everybody currently shipping an MCP server has a number. Ninety percent fewer tokens. Half the context. Three times faster retrieval. The numbers are usually real, in the narrow sense that somebody measured something and wrote it down.

I spent a couple of days and about $240 of API budget trying to produce one of those numbers honestly, for a tool I wrote myself. I ran 510 headless agent sessions across three separate benchmark designs. Almost none of it survived contact with scrutiny, and the parts that failed were more interesting than the parts that worked.

This post is about why measuring an agent tool is genuinely hard, in ways that are not obvious until you have already published. It is not a post about my tool. It is a post about the specific ways an evaluation slides, without anybody acting in bad faith, into proving nothing at all or proving exactly what its author hoped.

Disclosure: the tool under test is enola, which I wrote. That is the whole reason this post exists: I had every incentive to produce a flattering number, I designed three benchmarks in a row that could not have produced one, and it took an outside voice to point out why.

The number everybody reaches for is the wrong one

The instinctive benchmark for a context tool is payload size. Ask the tool a question, ask grep the same question, compare the bytes. It is easy, it is repeatable, and it produces a big percentage.

JetBrains published the cleanest warning about this I have seen. In a post testing whether terse prompting saves tokens, they found an advertised 65% saving collapsed to 8.5% once measured in a paired agentic benchmark rather than in isolation. Their own context product is evaluated on whole task execution: turns, latency, execution cost, completion. Not tool output size.

The reason payload size misleads is that a tool response does not cost you tokens, it replaces an exploration loop. Measuring the response instead of the thing it displaced rewards tools that answer less. A one-line answer wins every payload benchmark and might be useless.

So I skipped that. I went for accuracy instead: can an agent answer structural questions about a codebase more correctly with the tool than without it? That felt rigorous. It was the first mistake.

Three runs, and what each one actually measured

The design looked reasonable. Six questions about real open-source repositories. Several arms: an agent with no tool, an agent with the tool available, an agent instructed to use it. Five trials per cell, fresh session each time, execution order shuffled so that provider-time effects could not line up with a condition. Answer keys built and frozen before any session ran, each graded by an outside oracle such as gopls, tsc or go build, never by the tool under test.

That last constraint felt like the most important one. It turned out to be the fatal one, for reasons I will get to.

Run one, 150 sessions. Every single question-and-arm cell returned five byte-identical answer sets. Self-agreement of 1.000, everywhere. I had gone in expecting to demonstrate that agents are inconsistent, that asking the same question twice gives you two different answers. They were perfectly consistent. Different sessions, hours apart, different turn counts, identical output.

Then two of my six answer keys turned out to be wrong, and in both contested cases the agents had been more correct than the key that graded them.

Run two, 180 sessions. Rebuilt keys, fixed the arms. This run found something real: a genuine bug in my own tool, which I will come back to, because it is the one part of this exercise that paid for itself.

Run three, 180 sessions. Bug fixed, everything re-run. Accuracy came out identical across every arm, and the reason is less flattering than parity: the arms that merely had the tool available never once called it. Availability changed neither the answers nor the cost, because nothing was using it. The only arm that genuinely exercised the graph was an artificial one I had invented, described further down, whose numbers turn out to describe a configuration nobody runs.

Three runs. The honest summary of all three is: I proved that grep is good at grep-shaped questions.

The oracle is a participant in your experiment

Here is the part I would most want someone else to avoid.

I insisted on mechanical, third-party oracles so that nobody could accuse me of grading my own homework. gopls for references, tsc for type errors, go build for blast radius. Unimpeachable, all of them.

But think about what those tools can answer. They answer statically resolvable questions inside a single repository. Which is the same class of question a text search approximates well.

Requiring a mechanical oracle silently selects the questions your baseline is good at. You build a retrieval benchmark by accident, and then discover that retrieval tools are good at retrieval. Every question I wrote had an answer that was a string sitting in one repository. Of course grep found it. My design guaranteed the result before a single session ran.

That is the trap in one sentence, and it generalises well past my situation. If the only questions you can grade are the questions your baseline handles, your benchmark cannot distinguish anything.

Your ground truth is probably wrong in a direction that flatters you

Two of my six answer keys were defective, and the failure modes are worth naming because neither was carelessness.

The compiler answers a different question than you think. I built a “what breaks if I change this” key by modifying an interface and running go build ./..., then recording the files it rejected. Four files. Definitive, surely.

It is wave one. A package that fails to compile masks its own dependents, which are never type-checked at all. Fixing the four and rebuilding surfaces a fifth file. Running go vet, which type-checks tests, surfaces a sixth. A seventh is still hidden behind those. The compiler tells you what fails first, not what must change. Agents answering “six” were closer to the truth than my key of four, and I scored them at 0.80 for it.

Your question is ambiguous and the agent will notice before you do. Another question asked which files call a specific function. One agent returned eleven of twelve and explained itself: the twelfth file is inside the function’s own package, so it calls the function unqualified, and therefore does not contain the literal string I had named in the question. That is a defensible reading. My key said twelve. The disagreement was mine, not the agent’s.

Both of these flattered nobody in particular, but notice the shape: a wrong key produces a confident number, and nothing in the pipeline complains. If the error had gone the other way, I would have published it.

The harness lies to you, in six specific ways

Every one of these produced a wrong number before I caught it. If you evaluate agent tooling, expect all six.

Global tool configuration leaks into your control group. Coding agents load the user’s own configured MCP servers by default. My “no tools” baseline silently inherited a running instance of the very tool I was testing. A control group is not a control group until you explicitly pass an empty configuration and verify it took.

The delivery channel is part of the experiment, and the obvious channel is wrong. I delivered instructions through a command-line system-prompt flag. The actual product writes them into the files agents already read. Those are different channels with different adherence, so an entire run measured a mechanism that ships to nobody. Before trusting a null result, verify the channel is live: put a canary instruction in it, such as “begin every reply with PINEAPPLE”, and check the reply.

The harness has its own system prompt, and it outranks yours. With the real installer run, the session hooks firing, and the instruction placed directly in the user’s own message, agents reached for the tool in 3 of 120 sessions. That is a genuine finding about tool adoption, but it is a finding about that harness. A null adoption result inside one coding agent is not a fact about agents in general, and separating those requires a second harness with no system prompt of its own.

Subagents do not inherit tool restrictions. If your design restricts which tools an arm may use, a subagent spawned inside that session starts clean and can use everything you just forbade. Sessions escaped my restrictions this way in two separate runs.

git status cannot see side effects in ignored paths. Repositories commonly gitignore exactly the directories that agent tooling writes into. My cleanliness check reported all-clear over real residue. Snapshot the paths before and compare after, rather than asking git.

Build oracles fail silently in the direction of success. One repository would not resolve its dependencies under the module proxy setting on my machine, so the build died before type-checking and produced an empty error list. An empty error list looks exactly like a clean build. Any oracle whose failure mode is “no output” needs a positive check that it ran at all.

Then there is the arm I invented that nobody would ever run

My centrepiece arm removed the agent’s own tools. No grep, no glob, no shell. The tool under test had to answer everything.

I told myself this was a capability ceiling: what can the tool deliver when the agent cannot fall back? It is not. It is a straitjacket that no user has ever worn, and it quietly reframed the whole exercise as tool versus agent rather than tool inside agent. Every number that arm produced describes a configuration that does not exist.

This is an easy mistake to make and a hard one to see, because the crippled arm is the only one where your tool gets used at all, so it is the only arm that appears to be measuring anything.

Variance, and why one run of anything is worthless

Two published results are worth holding onto here. A study of coding-agent harnesses reports up to 40x token differences from harness choice alone, with model and task otherwise held constant. And an analysis of agent trajectories found repeated executions of the same task varying by as much as 30x in token consumption, with more tokens not reliably producing better outcomes.

Put those together and single-run comparisons of anything agentic are noise. So are means. Report medians with quartiles, run every cell several times, and shuffle execution order so that a slow afternoon on the provider’s side does not become your treatment effect.

I did all of that, correctly, on three benchmarks that were measuring the wrong thing. The statistics were the easy part.

What actually survived

Two things, and neither is the thing I set out to produce.

The benchmark worked as a defect detector. In run two, one question came back materially worse in a single arm, and it was the artificial one, the arm I have just spent a section calling invalid. That is not a coincidence: it was the only arm that ever put weight on the tool, so it was the only arm that could feel the tool break. Chasing it down uncovered a real bug: a configuration parser that recorded path aliases only in their wildcard form and silently dropped the exact form, which is how monorepos name sibling packages. The effect was that every cross-package import resolved to a phantom node, so the two most valuable queries in the tool returned nothing at all on a monorepo. On one repository this accounted for 4,847 call edges pointing at symbols that did not exist.

That bug had survived a 72-repository benchmark corpus, because it is invisible on any repository with a single package. A benchmark that keeps finding real bugs in the product is worth running even when it declines to produce a marketing number.

A deterministic artifact can be proven wrong. This is the part I find genuinely interesting. The bug was catchable because the tool makes a falsifiable claim: it emits a graph, and the graph can be checked against a compiler and shown to disagree with it. A mental model that an agent assembled from twenty greps cannot be audited, because it was never written down anywhere. It produces the same confident eleven-of-twelve, and nothing contradicts it.

That is a real argument for structural tooling, and notice that it is not the argument I was trying to measure. It arrived sideways, out of a failed benchmark.

If you are about to benchmark an agent tool

The short version of two days:

1. Measure task outcomes, not tool payloads. What did it cost to correctly complete the work? Cost per successful task beats tokens per call, and it stays honest when your tool costs more but fails less.
2. Check whether your baseline can already answer your questions. If a text search handles them, you are measuring text search. Write the question first, then find an oracle, never the reverse.
3. Never cripple an arm. Compare the full agent to the full agent plus your tool. Anything else measures a product nobody uses.
4. Assume your ground truth is wrong until something independent disagrees with it and you have worked out which of you is right.
5. Verify every channel and every oracle actually fired. Silence is not a result.
6. Publish where it does not help. A benchmark with no negative rows is a brochure.

And the uncomfortable one, which is really the whole post: a benchmark you designed for a tool you built will find what you were looking for unless you build it so it can fail. Mine could not fail for three straight attempts, and every individual decision along the way looked like rigour at the time. Mechanical oracles, frozen keys, shuffled order, repeated trials, medians rather than means. All correct. All in service of a question that could only have one answer.

The tell, if you want one: if you cannot describe in advance the result that would make you abandon your tool, you are not running a benchmark.