AI Agents
Your Agent Needs a Router, Not a Single Model
One model call per turn is simple until it is slow, rate-limited, or expensive. Routing is the fix and it is not hard.
The first version of my coding agent called one model. Every prompt, one provider, one key, one model. It worked, and then it was slow on a large refactor, and the failure mode was that there was nothing to fall back to.
The fix was not a better model. It was a router.
Why one model per turn does not scale
Three different kinds of request arrive in the same stream: a file search, a one-line edit, and an architecture argument that needs a long context. They have wildly different cost and latency profiles and no reason to share a configuration.
- A cheap, fast model handles tool calls, file reads, and classification.
- A strong model handles planning and anything that changes several files at once.
- A fallback exists for when the strong one is rate-limited, which is always.
Routing on purpose, not on vibes
The routing rule is the whole design. Mine is intentionally boring: quota first, then prompt shape, then a probe of what the provider is actually offering right now.
1. M1 quota - is there budget left for this turn?
2. M2 probe - what is the upstream provider healthy right now?
3. M3 health - which configured routes are actually up?
4. M4 policy - pick the strongest route that passed the above
5. M5 fallback - on failure, demote and try the next viable routeThe part that matters is the outcome hook. A fallback that only triggers on an exception misses the case that actually hurts: a provider returning 200 with degraded output, or a stream that dies halfway through. The router has to learn from the result of every call, not just from whether it threw.
What this buys
Predictable cost on the common path, real headroom when the strong model is unavailable, and — the part I care about most — a single place where a provider degradation is visible instead of being absorbed silently across the codebase.