AI Agents
Verify the Output Before You Ship the Agent
Every agent I have shipped needed the same unglamorous layer: a checker that can say no.
Every agent I have shipped needed the same unglamorous layer: something that can say no. Not a test suite — a checker that runs after the agent says it is done, and has the authority to send it back.
This is not an optimisation. It is the difference between a demo and something you can point a colleague at.
An agent will confidently finish a broken task
The failure is rarely the code. It is the report. An agent that changed three files and could not apply the third will say the change is complete, because from its side the turn ended normally.
So the verifier's first job is not correctness. It is checking that the claimed outcome happened.
What to verify, in order of value
- The build passes. Cheap, and it catches the majority of real failures.
- The specific behaviour the task claimed changed, actually changed — assert on it.
- Nothing outside the stated scope was touched. Agents widen silently.
- Tests that existed before still pass. This is the one people skip and regret.
Scope drift deserves its own check
An agent asked to fix one thing will sometimes 'improve' something adjacent, and it will describe this as part of the fix. So compare the diff against the task, not just against the tests.
verify(task, diff):
1. build passes? -> else retry
2. task assertion holds? -> else retry
3. every changed file in scope? -> else flag, do not retry blindly
4. pre-existing tests pass? -> else retryGive it real authority
A verifier that cannot fail the run is a comment. The whole value is in the loop back: the checker rejects, the agent sees why, and it tries again with information. Remove the authority and you have written a very expensive log file.