F3 · Model cascade and routing
Family: Oversight and operations
Each request goes to the cheapest model able to handle it, and moves up a notch if confidence is too low.
Principle
The request goes first to a small model, and moves up to a larger one only when confidence is too low: that is the cascade. A learned router can also pick the right model directly. Other variants: a strong model leads and light models execute, or a weak model consults a strong one when needed.
The problem it solves
Sending every request to the largest model is expensive, while a large share of them are simple.
When to use it
- Traffic mixes simple and hard cases, and you have a reliable judge or confidence score.
- An evaluation set lets you calibrate the threshold for moving up to the large model.
When to avoid it
- There is no reliable judge and no evaluation set: the escalation threshold cannot be calibrated.
- Gains depend heavily on the task (−85% on MT-Bench, but −45% on MMLU with RouteLLM): measure on your own data before committing.
Common pitfalls
The anti-patterns that misuse this pattern, best established first. Each sheet describes the symptoms, the public incidents and the remedies.
Framing and choice of level
- AP05 · One model for everything — The same model serves every step, from trivial reformatting to hard reasoning.
Instructions
- AP24 · Legacy over-prompting — “CRITICAL: You MUST” lines written for the old model, which the new one takes literally.
Context and memory
- AP30 · Cache-busting prefix — A timestamp or a reordered JSON at the top of the prompt, and every call pays full price for the whole context again.
Oversight, operations and governance
- AP77 · Silent regressions on change — The model, the prompt or the infrastructure changes, and only users notice the drop in quality.
Synonyms by vendor
- Model cascade (FrugalGPT)
- Learned router (RouteLLM (LMSYS))
- Smart friend (Cognition (2026))
- Single-agent then multi-agent cascade (Gao et al. (2025))
Neighbouring patterns
Sources
- FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance — arXiv, 2023-05
- RouteLLM: An Open-Source Framework for Cost-Effective LLM Routing — LMSYS, 2024-07-01
- RouteLLM: Learning to Route LLMs with Preference Data — arXiv, 2024-06
- Single-agent or Multi-agent Systems? Why Not Both? — Gao et al., arXiv, 2025-05
- AI Agents That Matter — Kapoor, Stroebl, Siegel, Nadgir, Narayanan (Princeton), arXiv, 2024-07
Content reviewed on 2026-10-02