# Shadow Evaluation Shadow evaluation means running a candidate model *next to* the one you have in production, on the same real inputs, while only the production model's answer is used. You log both outputs, compare them, and switch only when the numbers say the candidate is at least as good (or good enough for much less money). Why bother? Because benchmarks and vendor claims tell you how a model performs on *someone else's* data. The only thing that matters is how it performs on *yours*: your tickets, your invoices, your weird edge cases. Shadow evaluation answers that question without a single customer seeing a wrong decision. ## Where it comes from The idea is older than LLMs. It's a classic deployment technique (see [[ML Deployment Patterns]] and [[Continuous Deployment (CD)]]): - **Dark launches.** Facebook popularized the term when it launched Chat in 2008. For a while, Facebook pages connected to the chat servers, queried presence and simulated message sends "without a single UI element drawn on the page." The real load hit the new system before any user could use it - **Traffic mirroring.** Service meshes and proxies (Envoy, Istio) can copy live requests to a second service and discard its responses. Same idea at the network level - **Shadow models in MLOps.** A new fraud or ranking model scores live traffic for weeks, its predictions stored and compared, before it takes over (see [[MLOps]]) With AI models there's an extra twist: the outputs aren't just "worked or crashed". They're *decisions*, and two models can disagree on a decision without either one throwing an error. That's why the comparison step matters more than the plumbing. ## Online vs offline Two flavors, and I'd do both: - **Offline replay.** Take a log of past decisions whose correct answer you already know, and run the candidate on it. Fast, cheap, repeatable. The risk: last quarter's data may not look like next month's - **Online shadow.** Duplicate live traffic to the candidate for a few weeks. Slower, but it catches what replay misses (new input shapes, latency under real load, drift) ## What to compare - **Agreement rate.** How often do the two models give the same answer? High agreement tells you little about quality (they can be wrong together), but it tells you how much of the traffic is "easy" - **Disagreements, reviewed by a human.** This is where the value is. Sample the cases where the models disagree and have someone who owns the decision label them. Now you know who's right *when it matters*, and a small labeled set covers the interesting part of the distribution - **Accuracy on your own labels.** Against the final human disposition, not the old model's first guess (otherwise you're measuring how well the candidate imitates the incumbent) - **Calibration on your data.** Does 0.9 confidence mean right 90% of the time *here*? Build a reliability diagram (see [[AI Model Calibration]]) - **Cost per correct decision**, not cost per token. A model that's 10x cheaper but needs a fallback on 40% of cases may not be cheaper at all - **Latency**, at the percentiles you care about (p95, p99), under real load ## The Jev case Rajesh Beri's test of [[Jev]] (September 2026) is the best public example of why you shadow-evaluate a [[Decision Models (DMs)|decision model]] before trusting it: - On 2,000 phishing emails, a single "is this phishing?" question gave Jev **62.6%** accuracy, against **81.3%** for Claude Haiku 4.5. A naive swap would have been a disaster - Five decomposed yes/no questions ([[Atomic Question Decomposition]]) combined with a [[Logistic Regression|logistic regression]], fit on 1,000 emails and tested on the other 1,000, reached **95.0%** vs 93.2% for Haiku, at 12 to 27x lower cost ($0.038 per 1,000 emails vs $0.462 for Haiku) - On synthetic support tickets he measured an expected calibration error of **0.107** (Archer Hume measured 0.031 on MMLU). Same model, different data, different calibration - His whole pre-registered study was 5,721 calls for **$0.176** His conclusion: "The 95% is not Jev. It is Jev plus your labelled data plus a regression you maintain." Only an evaluation on your own labeled data can tell you which of those two numbers (62.6% or 95%) you'll actually get. ## Checklist: moving an LLM call to a decision model This is how I'd migrate a classification-style LLM call (routing, triage, tagging) to Jev or any other decision model: 1. **Pick one high-volume decision with ground truth.** Ticket routing, invoice categorization, alert triage. Beri suggests 1,000 to 2,000 records, labeled with the human's final decision 2. **Size the sample honestly.** At 90% accuracy, 1,000 examples give a 95% confidence interval of about ±1.9 points; 200 examples give about ±4.2. And if a class you care about is 2% of traffic, 1,000 records contain about 20 of them. Oversample rare classes on purpose 3. **Pin the model version.** Use `jev-1.13.0`, not `jev-latest`. Aliases move on every release, and your thresholds were tuned on a specific model. TypeSafe's docs say the same, and the response's `model` field tells you which version actually answered. Log it 4. **Write the questions like code.** Have the person who owns the decision review the instructions and criteria. Give every Choice a "none of these" option 5. **Log everything, act on nothing.** Input, production answer, candidate answer with its probabilities, latency, cost, model version 6. **Label the disagreements.** A human reviews a sample of the cases where the two models disagree. This is the cheapest high-value labeling you'll ever do 7. **Tune thresholds on half, test on the other half.** Fit per-question calibration or thresholds on one half, measure on the held-out half. Find the confidence band where accuracy clears your bar, and measure what share of traffic it covers (see [[Confidence-Gated Routing|routing on confidence]]). For a guarantee rather than an estimate, put a [[Conformal Prediction]] layer on top 8. **Attack the input.** Plant instructions inside the ticket or email and check whether the answer moves. Decision models read the state as data, but that doesn't make them immune 9. **Switch gradually.** Route only the confident band to the new model, keep the LLM as the fallback for the rest ([[AI Model Cascades]]), and keep a small shadow sample running after the switch to catch drift 10. **Re-run on every model change.** New version, new prompt, new category: shadow-evaluate again before switching ## Limits - **It needs labels to mean anything.** Agreement alone can't tell you which model is right - **The past isn't the future.** A replay on last month's data can't catch next month's drift. Keep a trickle of shadow traffic running - **Side effects.** A candidate that *acts* (sends email, calls APIs) can't be shadowed naively; stub the actions or evaluate only the decision - **Cost.** Online shadowing doubles inference for the duration. With a model at $0.042 per million input tokens that's negligible; shadowing a frontier LLM is not ## My take I think shadow evaluation is the single most useful habit for anyone putting AI into production, and the most skipped one. The Jev story shows why: the same model gives you 62.6% or 95% depending on how you use it, and no blog post (including this one) can tell you which one you'll get on your data. A few thousand logged decisions, a human reviewing the disagreements, a pinned model version. That's a weekend of work that replaces a lot of arguing about benchmarks. ## References - [TypeSafe Jev: typed decision model, calibration, decomposition, shadow eval (Rajesh Beri)](https://www.beri.net/article/typesafe-jev-typed-decision-model-calibration-decomposition-shadow-eval) - [Facebook Chat (Engineering at Meta, 2008)](https://engineering.fb.com/2008/05/13/web/facebook-chat/) - [Jev's Architecture Unmasked (Archer Hume)](https://archerhume.com/posts/jevs-architecture-unmasked/) - [Confidence (TypeSafe docs)](https://docs.typesafe.ai/confidence) - [Models and aliases (TypeSafe docs)](https://docs.typesafe.ai/models) ## Related - [[Jev]] - [[AI Model Calibration]] - [[Confidence-Gated Routing]] - [[Decision Models (DMs)]] - [[AI Evaluation]] - [[Conformal Prediction]] - [[Atomic Question Decomposition]] - [[AI Model Cascades]] - [[AI Model Selection]] - [[ML Deployment Patterns]] - [[MLOps]] - [[Continuous Deployment (CD)]] - [[Human-in-the-Loop]]