The software factory · model routing
Model routing for AI agents
One model for everything is the expensive mistake. A software factory runs a fleet of jobs with wildly different needs — and the cheapest correct setup routes each job to the smallest model that can actually do it. Here's the routing we run, the traps that bit us, and a sketch of the piece we haven't built yet: a router that upgrades itself.
"Which model should I use?" is the wrong question for an agentic system. The right one is "which model for this job?" — because an agent that orchestrates and decides has almost nothing in common, cost- or capability-wise, with one that bulk-edits files or one that writes a pull request. Point them all at a single frontier model and you overpay for the grunt work while underspending on the judgment. Route by job, and the bill drops by an order of magnitude while the quality where it matters goes up.
Below is the role taxonomy we actually route on. Tap a role to see the tier it maps to, why, and the trap that's waiting there.
Route by the job, not by the model
"Tier" is the class of model, not a brand — the specific models rotate constantly (see below). The tag shows where the money goes: a cheap paid model where judgment lives, free/local everywhere else.
Three traps that don't show up in benchmarks
The routing table looks obvious once it's drawn. Getting there cost us three lessons that no leaderboard warns you about:
1. The tool-call trap. For the local grunt tier we first reached for a coder-branded 7B model — top of the coding charts for its size. It silently edited nothing. The model had no tool-call template, so when asked to use a tool it printed the tool call as plain text and the harness saw an empty turn. The run reported "success" while changing zero files — the worst possible failure, a confident no-op. The fix reordered our selection criteria: for any agentic role, filter by tool-call support first, coding score second. A smaller general model with native tool calling beat the higher-scoring coder that couldn't hold a wrench.
Verify the diff, never the transcript. A model that "says" it made the change and a model that made the change look identical until you read the git diff.
2. The free-tier economics trap. Free frontier models are real and useful — but for an agentic loop, throughput is dominated by fixed per-request overhead and rate caps, not by model size. A giant free model and a small free one finished a single step in nearly the same wall-clock, and with a large prompt on every step plus a low requests-per-minute cap, multi-step sessions read as slow and stall-y. That's the true price of $0. The lever that actually helps: put a cheap paid model on the control plane — pennies a day for the orchestration and judgment — and keep the grunt tier local and free. You're not buying capability, you're buying the absence of throttling on the one path where latency compounds.
3. The stale-roster trap. Every "best free model for coding" blog post is out of date by the time you read it — the free roster rotates on a scale of weeks. The model three articles agree is the one to use is frequently already gone. So don't hardcode a model list: query the provider's models endpoint at setup time and filter — by price tier, context window, and capability flags — instead of trusting anything a human (or a blog) wrote down last month. This single habit is what makes the next section possible.
The self-updating router — a design, not yet a shipped feature
If routing is "match the model to the job," the obvious next question is: who keeps the match current as new models appear every week? Doing it by hand means the factory is always running last month's routing table. So the design is an agent whose job is the routing table — it discovers, evaluates, proves, and proposes, and a human signs the change. Four stages. Tap through them.
The upgrade loop
The whole loop is designed to cost near-zero to run and to end at a human. New models are cheap to test; the only expensive mistake is silently trusting one.
Why it ends at a human
A model swap is not a reversible in-repo change — it silently reshapes the cost and behavior of every future task until someone notices. So it sits in the same place as any other consequential, hard-to-undo action in the factory: behind a human approval gate, with an evidence package attached. The agent does the tedious, continuous part — discovery, scoring, the live-fire A/B — and presents a recommendation a human can accept in one tap or reject. That's the pattern the whole factory runs on: the agent proposes and accelerates; a deterministic gate, or a person, decides. Self-improving is not the same as self-approving, and the gap between them is the entire safety story.
The hardest-won lesson from the one real run: an eval pipeline is code review times three. The baseline you compare against, the scorer that grades the output, and the comparator that picks a winner each fail silently in ways that produce confident nonsense — a saturated rubric will happily crown four "winners" that are all noise. A router that upgrades itself is only as trustworthy as the eval it upgrades against, which is exactly why the last stage is a human reading the evidence, not a cron job flipping a config.