AI Agents
Measure Your Agent as a System, Not a Model
A per-prompt benchmark hides the only number that matters: what the whole pipeline costs when it fails.
An agent is a pipeline: a router, a model call, a set of tools, a verifier, and usually a retry. Benchmarks measure one of those links and call it a system score.
That is fine for comparing models. It is useless for deciding whether to change anything.
The per-prompt number hides the cost of failure
A prompt benchmark answers: given this input, how good is this output? The operational question is different: given this input, what did the whole pipeline cost, and how often did it fail in a way that forced a retry?
cost(task) = first_attempt_cost
+ retry_rate * retry_cost
+ verification_cost
+ (escalation_rate * manual_intervention_cost)
Not measured by any single-call benchmark.A model that is 4% worse per call can be dramatically cheaper per task if it never escalates. A model that is faster per call can be much slower per task if it fails in a way that triggers a long backoff.
Measure the pipeline that actually ran, including the attempt that failed.
Instrument the boundaries, not the prompt
The useful measurements are at the seams:
- Which route was chosen, and what would have been chosen if quota were not exhausted.
- Tokens in, tokens out, and cache hit rate, per route rather than in aggregate.
- Verification pass or fail rate — the number that tells you if the agent is trustworthy at all.
- Retries and escalations, keyed by the error class that caused them.
One trap worth naming
Timing context is the classic mistake here. If you keep a module-level reference to a request's timer, two concurrent requests share it and you get spans from different turns mixed into one measurement. The numbers look plausible and they are wrong, which is worse than having no numbers.