# AI Model Cascades A model cascade sends every request to a cheap model first, decides whether that answer is good enough, and escalates to a more expensive model only when it isn't. Most inputs are easy, so most of them never reach the expensive tier. You get close to the big model's quality at a fraction of its cost. It sounds like [[Model routing]], and the two are cousins, but there's one difference I find important. A router decides *before* anyone answers ("this looks hard, send it to the big model"). A cascade decides *after* the cheap model answered ("this answer looks wrong, try again higher up"). A router predicts difficulty; a cascade checks work. The quality of a cascade depends entirely on how good that check is. ## The general idea The classic cascade is the Viola-Jones face detector (2001). Scanning an image means evaluating a huge number of sub-windows, and almost none contain a face. So the first stage is a tiny classifier (two features) tuned to almost never miss a face while throwing away a large share of the windows. Each following stage is more expensive and only sees what survived. That's what made real-time face detection possible on 2001 hardware. LLMs brought the idea back: - **FrugalGPT** (Chen, Zaharia, Zou, Stanford, 2023) chains LLM APIs from cheap to expensive, with a learned scorer that decides whether to accept each answer or move up. The paper reports matching GPT-4's performance with up to 98% cost reduction, or improving on GPT-4's accuracy by 4% at the same cost - **RouteLLM** (LMSYS, 2024) is the router side of the same coin: classifiers trained on Chatbot Arena preferences pick a strong or weak model per query. Over 85% cost reduction on MT Bench (45% on MMLU, 35% on GSM8K) against GPT-4 alone, while keeping 95% of GPT-4's quality - **[[AI Speculative Decoding]]** is a cascade at the token level: a small draft model proposes tokens, the big model verifies them, and you only pay the big model's full price where the draft was wrong The weak point of every LLM cascade is the scorer. How do you know the cheap answer is wrong? Asking another LLM "is this answer good?" adds latency and cost, and a vague "is this good?" gives mushy answers. Model-written confidence ("I'm fairly sure") isn't a probability you can threshold. ## How Jev makes it cheap [[Jev]] is a good fit for the scorer role for three reasons: it's fast (~100 ms), it's cheap enough that the check doesn't eat the savings ($0.042 per million input tokens, output free), and it returns probabilities you can threshold. TypeSafe also pitches it for the router role (a Choice over which model to call), which is [[Model routing]] done with a decision model. ## Worked example: the SDE cascade The structured-data extraction (SDE) cookbook builds a three-step cascade: 1. **Extract** with `gpt-5.4-mini` ($0.75 / $4.50 per million tokens) 2. **Verify** each field with a battery of Noul questions in one Jev call 3. **Escalate** to `gpt-5.5` with high reasoning effort (~7x the mini's price) only if a verifier fires The walkthrough uses an NYU events-calendar page where the scrape captured only navigation and boilerplate. There's no registration date and no description on the page. The mini model returns a *schema-valid fabrication*: `description: "Registration opens for the fall semester"`, which is the example value from the schema's own field description, parroted back. JSON Schema validation passes. [[LLM Structured Outputs]] wouldn't catch this either, because the structure is fine; the content is invented. The Jev verifier asks narrow per-field questions, each framed so that TRUE means "something is wrong". Probability of a problem: | Verifier head | P(wrong) | |---|---| | hallucinated | 0.95 | | off_target | 0.85 | | unreasonable | 0.58 | | whole-record "should this record be escalated?" | 0.56 | | incomplete | 0.16 | | format | 0.10 | | type mismatch | 0.02 | Two heads fire above 0.7, so the record escalates. The reasoning model returns an honest empty `""`. Notice the whole-record judge: 0.56, basically a shrug. The per-field heads localize the problem; the "is the whole thing good?" question doesn't. Over 100 prompts (TypeSafe's internal results, shown as a chart), sweeping the escalation threshold from 0 to 1 traces a Pareto frontier that sits up and to the left of every single model. `gpt-5.5` alone scored about 0.81 quality at about $0.10 per extraction; the cascade gets most of that quality for a fraction of the cost. ## What makes a good verifier The cookbook's appendix is the most reusable part, and it applies to any cascade: - **Narrow and grounded.** One checkable fact about one field against the source ("is this value absent from the source?"), not "is this extraction good?" - **Bad = TRUE.** Phrase every question so the escalate case is the yes case, with explicit criteria for what true and false mean - **Per field, then aggregate with max.** One confident red flag should escalate. Averaging would drown it ("averaged into silence") - **Independent of the extractor.** A different model catches blind spots the extractor shares with itself - **Cheap.** If the check costs as much as the big model, there are no savings left - **Separating.** High on real errors, low on correct fields, so one threshold splits accept from escalate cleanly ## Gotchas - **The Pareto chart is internal and unreplicated.** TypeSafe says the costs weren't recalculated at the current Jev rate. Treat it as a demonstration, not a benchmark - **Stochastic cheap models.** The mini model invented a different description on nearly every run, even at temperature 0. The cookbook hard-codes one fabrication for reproducibility. Your cascade has to hold up against that variability - **The verifier can only flag what it's asked about.** A missing head means a missed error class. Spend time on the battery - **Latency adds up on escalation.** The escalated items pay cheap model + verifier + expensive model. Fine for batch, worth measuring for interactive paths - **Calibration drifts.** The threshold you tuned last month may not hold on next month's data. Re-check the frontier periodically ## My take Cascades are one of the few AI cost-cutting techniques I trust, because they don't pretend the cheap model is as good as the expensive one. They just stop paying for the expensive one when it isn't needed. The [[Pareto Principle]] is at work: most inputs are easy, and a small share of hard cases deserves the big model. What the TypeSafe cookbook adds is a concrete answer to the hardest part, the scorer. Break verification into narrow, per-field, bad-equals-true questions, OR them together with max, and you get a check that's cheap and points at the exact field that's wrong. I'd use the same design with any verifier, decision model or not. And I'd combine it with [[Confidence-Gated Routing]] at the end of the chain: when even the big model is unsure, a human should see it. ## References - [SDE cascade cookbook (TypeSafe docs)](https://docs.typesafe.ai/cookbooks/sde_cascade) - [Use-case map (TypeSafe docs)](https://docs.typesafe.ai/concepts/use-case-map) - [Chen, Zaharia, Zou, FrugalGPT (2023)](https://arxiv.org/abs/2305.05176) - [Ong et al., RouteLLM (2024)](https://arxiv.org/abs/2406.18665) - [RouteLLM announcement (LMSYS blog, 2024)](https://www.lmsys.org/blog/2024-07-01-routellm/) - [Viola-Jones object detection framework (Wikipedia)](https://en.wikipedia.org/wiki/Viola%E2%80%93Jones_object_detection_framework) ## Related - [[Model routing]] - [[Confidence-Gated Routing]] - [[AI Speculative Decoding]] - [[AI Cost Management]] - [[AI Hallucination]] - [[LLM Structured Outputs]] - [[Atomic Question Decomposition]] - [[Pareto Principle]] - [[Jev]] - [[Decision Models (DMs)]]