Home / Rank #3
Model routing & cascades
Most requests do not need your most expensive model. Routing sends simple queries to cheap models (Haiku, GPT-mini class, Nova) and reserves frontier models for the requests that actually need them; cascade designs try cheap first and escalate only on failure.
Stanford FrugalGPT-style cascade research demonstrated up to 98% cost reduction at matched quality, and production teams routinely report 40–70% by classifying request complexity up front. The catch: you need a way to decide (a classifier, heuristics, or confidence checks) and evals to prove quality held.
How to do it
- Segment traffic by task type; label which segments a small model already handles well.
- Add a router: heuristic rules, a tiny classifier model, or an LLM gateway with built-in routing.
- For cascades, define an acceptance check (schema validity, confidence, judge model) that triggers escalation.
- Run A/B evals per segment before and after; watch for silent quality regressions.
Frequently asked questions
What is the difference between routing and a cascade?
Routing decides the model before the call based on the request. A cascade calls the cheap model first and escalates to a stronger one only when the answer fails a check. Cascades save more but add latency on escalated requests.
Tools for this method
LiteLLM
The default self-hosted gateway: one OpenAI-compatible proxy across 100+ providers with budgets, caching, routing, fallbacks, and …
Portkey
Managed gateway with the strongest built-in semantic caching, plus guardrails, routing, and cost analytics. The low-ops route to t…
OpenRouter
One API and one bill across hundreds of models from every major lab — the fastest way to arbitrage the model price war and A/B che…
Next method: #4 Semantic caching