Model routing

Using your most capable (most expensive) model for every request is like sending a senior engineer to reset every password. Model routing matches the model to the job — most requests are easy and a small fast model nails them; a minority are hard and deserve the frontier model. Done well, routing cuts cost dramatically with little quality loss, because it spends capability only where capability is needed.

Routing strategies

• Route by task type. Sub-jobs have different needs: query rewriting, summarization for compaction, and simple classification don't need your flagship — a small model is faster and cheaper and just as good. You've already been doing this instinctively in RAG and Agents; make it explicit.
• Route by difficulty. A cheap upfront classifier (or a heuristic — input length, category, keyword flags) sorts requests into easy/hard, and each goes to the right tier. The classifier's cost is tiny against the savings.
• Cascade (escalate on failure). Try the cheap model first; if its output fails validation or its self-reported confidence is LOW, escalate the same request to the stronger model. Most requests resolve cheaply; only the ones that need it pay for the big model. This reuses the confidence signals you already compute.

Choosing the smallest model that clears the bar

The disciplined way to pick a model for a job is not vibes — it's your eval harness. Run the golden set against each candidate model and read the quality/cost/latency table. Frequently a smaller model clears your quality bar at a fraction of the cost, and the eval proves it rather than you guessing. The bar, not the leaderboard, decides: the best model for a job is the cheapest one that passes.

• Re-evaluate on model updates. New model versions shift the table — a small model that used to fail might now pass, or a routing tier might need retuning. Your CI eval (Module 6) catches this automatically.
• Routing adds a failure surface — the router itself can misroute. Keep it simple, and let the cascade's escalation be the safety net for a misjudged 'easy'.
• Log the model per request (your traces do) so you can see the routing mix and its cost/quality by tier.