Route Models by Task, Not by Reputation

A coding benchmark, a reasoning leaderboard, and a chat preference score may each be valid while predicting little about classifying one organization's planning intents or extracting fields from one document type. Model choice becomes operational only after the task and its failure cost are named.

A strong routing eval uses representative inputs, repeated runs, an explicit quality floor, latency, and full cost including review. The best model is the cheapest option that clears the floor reliably, not the model with the highest aggregate reputation. Reputation is what gets purchased when the job has not been measured.

Classification, normalization, retrieval ranking, and structured extraction often have narrow output contracts. Small or local models can perform well when the ontology is stable and failures are easy to detect. Frontier models remain useful where ambiguity, synthesis, or adversarial judgment dominates.

The boundary should be measured rather than assumed. A local model that misses a rare but costly intent may be more expensive than a paid model after escalation and rework. Error severity belongs in the routing score. Average accuracy will hide the class of mistake that actually creates a ticket.

A low-cost route needs a way to decline uncertain work. Confidence scores alone are unreliable, so abstention can combine schema validation, disagreement across repeated runs, out-of-distribution checks, and task-specific invariants. Failed checks escalate to a stronger model or a person.

This design converts model selection from a static configuration into a policy: attempt the cheapest eligible route, verify the result, and spend more only when the task earns it. Without abstention, routing to the small model is a cost cut that later appears as review load.

Routing results decay when prompts, schemas, tools, or source data change. A model that cleared last quarter's task may fail after an ontology expansion. The eval set should version with the work and include recent production disagreements, which are the cases humans already had to correct.

The practical dashboard is small: pass rate by task class, severe-error rate, abstention rate, latency, and cost per accepted result. Everything else is supporting detail. If severe errors are not visible separately from overall pass rate, routing will optimize for vanity.