# Conformal Prediction Conformal prediction takes any model that outputs scores or probabilities and turns its predictions into **sets** with a guarantee attached. Instead of "the answer is *billing*", you get "the answer is in {billing, refund}", and the promise is that over many predictions, the right answer lands inside the set at least 90% of the time (or whatever level you pick). The part I love: the guarantee doesn't depend on the model being any good or being well calibrated. It only needs a few hundred labeled examples from the same kind of data you'll see in production. The model can be a random forest, a fine-tuned BERT, an LLM or [[Jev]]. Conformal prediction doesn't care. The framework comes from Vladimir Vovk, Alex Gammerman and Glenn Shafer (*Algorithmic Learning in a Random World*, Springer, 2005). The most readable entry point by far is Anastasios Angelopoulos and Stephen Bates, *A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantification* (arXiv:2107.07511, 2021), which comes with Python notebooks. ## What the guarantee says If your calibration data and your future data are **exchangeable** (roughly: drawn from the same distribution, in no particular order), then for a chosen error rate α: P(true label ∈ prediction set) ≥ 1 − α Three words in there matter a lot: - **Distribution-free.** No assumption about the shape of the data or about the model - **Marginal.** The 90% holds *on average over all inputs*, not for every single input or every subgroup (more on that below) - **Finite-sample.** It holds for your actual calibration set size, not only "as n goes to infinity" ## How split conformal works (with a mini example) The simplest version is **split** (or *inductive*) conformal prediction. You split off a calibration set the model never trained on, then: 1. **Pick a nonconformity score.** It measures how "strange" a label looks for an input. For classification, the classic choice is s = 1 − p(true class). If the model gave the right answer 0.9, the score is 0.1 (not strange at all). If it gave it 0.05, the score is 0.95 (very strange) 2. **Score the calibration set.** For each labeled example, compute s using its *true* label 3. **Take a quantile.** With n calibration examples and target error α, sort the scores and take the ⌈(n+1)(1−α)⌉-th smallest one. Call it q̂ 4. **Build sets.** For a new input, include every label whose score is ≤ q̂, i.e. every label with p(label) ≥ 1 − q̂ A concrete case. I classify support tickets into billing, refund, shipping or other, and I want 90% coverage (α = 0.1). I label 500 recent tickets and score them. ⌈501 × 0.9⌉ = 451, so q̂ is the 451st smallest score. Say it comes out at 0.72. The rule becomes: **keep every label with probability ≥ 0.28**. - Ticket A: billing 0.91, refund 0.06, shipping 0.02, other 0.01 → set = {billing}. Automate it - Ticket B: billing 0.48, refund 0.41, shipping 0.08, other 0.03 → set = {billing, refund}. The model is torn, and the set says so. Send it to a human (or a bigger model) with the two candidates - Ticket C: four labels around 0.25 each → empty set. Nothing looks plausible enough. That's a signal the ticket doesn't look like anything in the calibration data That's it. A sort, a quantile, a threshold. The math that justifies it is subtle; the code is ten lines. ## Set size is the useful signal The size of the set is an uncertainty measure you can act on: - **Size 1**: confident enough to automate - **Size 2 or 3**: route to a human with a shortlist (a much easier job than choosing from scratch) - **Empty or huge**: something's off (new kind of input, drift, a question the model can't answer) That maps directly onto [[Confidence-Gated Routing|routing on confidence]], with one big difference: the threshold now comes with a coverage guarantee instead of a gut feeling. The score you pick controls the *shape* of the sets, never the guarantee. The "1 − p(true class)" score (called LAC) gives the smallest sets on average but can produce empty ones. APS (Adaptive Prediction Sets, Romano, Sesia and Candès, 2020) adds labels in order of probability until the cumulative mass passes a threshold, which adapts better to hard inputs. Same guarantee, different trade-offs. ## Conformal prediction for LLMs The obvious use: multiple-choice questions where you can read a probability per option. Kumar et al. (*Conformal Prediction with Large Language Models for Multi-Choice Question Answering*, arXiv:2305.18404, 2023) did exactly that with LLaMA-13B on 16 MMLU subjects and the 1 − softmax score. Two findings stuck with me: - Set size correlates tightly with accuracy, so it works as a selective classification signal (answer only when the set has one element) - When they calibrated on one subject and tested on another, coverage dropped below target (calibrated on high school computer science and tested on business ethics, coverage was around 83% instead of 90%). Exchangeability matters, and it breaks as soon as the data changes Later work extends it to free text (*Conformal Language Modeling*, Quach et al., 2023) and to API-only models without logits (arXiv:2403.01216), but the multiple-choice case is where it's cleanest. And that's exactly the shape of a [[Decision Models (DMs)|decision model]] output. ## Why it matters for Jev The biggest criticism of [[Jev]] is about calibration. TypeSafe says its probabilities are calibrated, and Archer Hume measured an expected calibration error (ECE) of 0.031 on MMLU. Rajesh Beri measured **0.107** on synthetic support tickets, with yes/no answers underconfident and multi-option answers overconfident. Same model, different data, different calibration (see [[AI Model Calibration]]). Anthony Maio adds that distribution shift can quietly break probabilities that were fine last month. Conformal prediction sidesteps the whole debate. You don't need Jev to be calibrated on *your* data. You need its probabilities to *rank* labels sensibly, plus a few hundred of your own labeled cases. Wrap a Choice in a conformal layer and you get a guarantee that rests on your data, not on the vendor's claims. The same goes for GPT, Claude, [[SemIf]] or anything else that returns a number per option. ## Limits - **Marginal, not conditional.** "90% on average" can hide 99% on easy tickets and 60% on the rare, hard ones. If a subgroup matters (a language, a customer tier, a rare class), calibrate per group (Mondrian conformal prediction) or check coverage per group - **Exchangeability breaks under drift.** New products, a new ticket form, a model version change: the calibration set no longer represents reality and the guarantee is gone. Recalibrate on fresh labels regularly, and pin the model version (an alias like `jev-latest` can change under you) - **Coverage is not usefulness.** A set containing every label always has 100% coverage. The guarantee is only as good as the model's ability to keep sets small - **It needs labels.** A few hundred is fine for a 90% target. For 99%, you need more, because the quantile sits in the tail - **It doesn't fix composition.** A guarantee on each question doesn't carry over to a workflow that combines ten questions with weights and branches. You'd calibrate the final decision instead ## My take I think conformal prediction is the most underused tool in applied AI. It's old (2005), it's simple, and it answers the question every team deploying a classifier should ask: "how often will this be wrong, on MY data?" Instead of arguing about whether a vendor's probabilities are trustworthy, I'd rather spend an afternoon labeling 500 cases and computing one quantile. That's a promise I can keep, and it keeps working when I swap models (as long as I recalibrate). ## References - [Algorithmic Learning in a Random World (Vovk, Gammerman and Shafer, Springer, 2005)](https://doi.org/10.1007/b106715) - [A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantification (Angelopoulos and Bates, arXiv:2107.07511)](https://arxiv.org/abs/2107.07511) - [Conformal Prediction with Large Language Models for Multi-Choice Question Answering (Kumar et al., arXiv:2305.18404)](https://arxiv.org/abs/2305.18404) - [Classification with Valid and Adaptive Coverage (Romano, Sesia and Candès, arXiv:2006.02544)](https://arxiv.org/abs/2006.02544) - [Conformal Language Modeling (Quach et al., arXiv:2306.10193)](https://arxiv.org/abs/2306.10193) - [API Is Enough: Conformal Prediction for Large Language Models Without Logit-Access (arXiv:2403.01216)](https://arxiv.org/abs/2403.01216) - [TypeSafe Jev: typed decision model, calibration, decomposition (Rajesh Beri)](https://www.beri.net/article/typesafe-jev-typed-decision-model-calibration-decomposition-shadow-eval) - [Jev's Architecture Unmasked (Archer Hume)](https://archerhume.com/posts/jevs-architecture-unmasked/) ## Related - [[AI Model Calibration]] - [[Confidence-Gated Routing]] - [[Jev]] - [[Decision Models (DMs)]] - [[Proper Scoring Rules]] - [[Shadow Evaluation]] - [[Human-in-the-Loop]] - [[Machine Learning (ML)]]