# AI Model Calibration A model is calibrated when its probabilities mean what they say. Take every prediction it made with 80% confidence: about 80% of them should turn out right. Same for 20%, same for 99%. Formally, P(correct | confidence = p) = p. Calibration is a property of a *group* of predictions. It says nothing about any single answer. A calibrated model that says 0.8 will still be wrong one time in five, and that's exactly what it promised. Why does this matter so much? Because a probability you can trust is something software can act on. You can set a threshold ("auto-approve above 0.9, send to a human below 0.6") and know roughly what error rate you're buying. With an uncalibrated model, 0.9 is just a vibe. This is the core pitch of [[Jev]] and [[Reinforcement Learning for Calibrated Decisions (RLCD)|RLCD]]. ## How you measure it - **Reliability diagram.** Bin predictions by confidence (0.0-0.1, 0.1-0.2...), plot the actual accuracy of each bin against its average confidence. Perfect calibration is the diagonal. Points below the diagonal mean overconfident, above means underconfident. The idea comes from weather forecasting (DeGroot and Fienberg, 1983) and was popularized in ML by Niculescu-Mizil and Caruana (ICML 2005) - **Expected Calibration Error (ECE).** The weighted average gap between confidence and accuracy across bins (Naeini, Cooper and Hauskrecht, AAAI 2015). Usually 10 to 15 bins. MCE (maximum calibration error) takes the worst bin instead - **[[Proper Scoring Rules]]** like the Brier score and log loss. They measure calibration AND sharpness together, which ECE doesn't ECE has real weaknesses. It depends on the binning. And a model that always predicts the base rate (say 0.3 for everything, on a dataset where 30% are positive) gets a perfect ECE while being completely useless. Calibration without *resolution* (the ability to separate cases) is worthless. ## Deep networks got overconfident Guo, Pleiss, Sun and Weinberger (*On Calibration of Modern Neural Networks*, ICML 2017, arXiv:1706.04599) showed that modern deep networks are much worse calibrated than older ones. A 5-layer LeNet from 1998 was nicely calibrated; a 110-layer ResNet was far more confident than accurate. Depth, width, batch normalization and less weight decay all made it worse. Their fix was almost embarrassingly simple: **temperature scaling**. Divide the logits by a single number T, fitted on a validation set. It doesn't change which answer wins (so accuracy stays the same), only how sharp the probabilities are. They called it "surprisingly effective". See [[AI Temperature]] for the same knob in LLM sampling. Older post-hoc methods include Platt scaling (1999) and isotonic regression (Zadrozny and Elkan, 2002). ## LLMs: calibrated at birth, broken by post-training Here's the part I find the most interesting. Pretrained language models are *well calibrated*, and post-training breaks it: - **Pretraining is calibrated by design.** Next-token cross-entropy is a strictly proper scoring rule: the loss is minimized only by predicting the true probabilities. So a base model's token probabilities tend to mean what they say - **Kadavath et al.** (Anthropic, *Language Models (Mostly) Know What They Know*, arXiv:2207.05221, 2022): "larger models are well-calibrated on diverse multiple choice and true/false questions when they are provided in the right format." RLHF policies looked badly miscalibrated because RL "tends to collapse language model predictions towards behaviors that receive the most reward". A single temperature of T = 2.5 largely fixed it across three evaluations - **The GPT-4 technical report** (arXiv:2303.08774, March 2023) measured it on an MMLU subset: ECE 0.007 for the pretrained model, 0.074 after PPO. In OpenAI's own words, "post-training hurts calibration significantly" - **Leng et al.** (arXiv:2410.09724) found that RLHF reward models prefer high-confidence answers, which pushes the policy toward overconfidence - **Kalai, Nachum, Vempala and Zhang** (OpenAI, *Why Language Models Hallucinate*, arXiv:2509.04664, September 2025) argue benchmarks that score right/wrong reward guessing over saying "I don't know", so models are "optimized to be good test-takers" Put together, it's a pretty damning story, and it comes from the labs themselves. Preference optimization (see [[Reinforcement Learning From Human Feedback (RLHF)|RLHF]]) rewards answers that *sound* confident. Binary correctness rewards ([[Reinforcement Learning with Verifiable Rewards (RLVR)|RLVR]]) pay a lucky guess like a sure answer. Neither rewards honest uncertainty. Almeida's version: "Overpromising is a feature... by design." ## Verbalized vs native confidence There are two ways to get a confidence out of an LLM: 1. **Native**: read the model's probability for each option (the logprob of "A", "B", "C", "D"). One forward pass, no generation. That's how the GPT-4 plot was made 2. **Verbalized**: ask the model to write "I'm 70% sure". Tian et al. (*Just Ask for Calibration*, EMNLP 2023, arXiv:2305.14975) found that for RLHF models, the verbalized number is often better calibrated than the token probabilities, precisely because RLHF distorted the latter Methods like RLCR (arXiv:2507.16806) and Rewarding Doubt (arXiv:2503.02623) train the verbalized number with proper scoring rewards. [[Jev]] goes the native route: it returns a probability distribution over typed options, and TypeSafe claims RLCD makes that distribution calibrated. Independent checks found an ECE of 0.031 on MMLU (Archer Hume) and 0.107 on synthetic support tickets (Rajesh Beri). Same model, different data, different calibration. ## Calibration vs discrimination Two things get confused all the time: - **Calibration**: do the numbers match frequencies? - **Discrimination** (or ranking): do confident predictions tend to be the correct ones? Measured with AUROC or a risk-coverage curve For [[Confidence-Gated Routing|routing on confidence]], discrimination matters at least as much. A threshold only helps if errors actually cluster at low confidence. A model can be perfectly calibrated on average and still rank its mistakes badly. ## Limits - **Calibration is relative to a distribution.** Calibrated on MMLU doesn't mean calibrated on your invoices. Distribution shift silently breaks it - **It doesn't compose.** Five calibrated yes/no answers combined with weights and thresholds don't give you a calibrated workflow (Anthony Maio's point about Jev) - **Per-question-shape differences.** Beri found Jev's yes/no answers underconfident and its multi-option answers overconfident on the same data - **The fix is cheap to check.** A few hundred labeled examples, a reliability diagram, maybe a temperature or isotonic fit on top. Or [[Conformal Prediction|conformal prediction]], which gives coverage guarantees without trusting the vendor's calibration at all ## My take I think calibration is the most underrated property in applied AI. We obsess over accuracy, but accuracy alone doesn't tell you *when* to trust the model, and that's the question automation depends on. The good news: it's measurable with a spreadsheet. If I were putting any model's probabilities behind a threshold, I'd build that reliability diagram on my own data first, whatever the vendor claims. ## References - [On Calibration of Modern Neural Networks (Guo et al., arXiv:1706.04599)](https://arxiv.org/abs/1706.04599) - [Language Models (Mostly) Know What They Know (Kadavath et al., arXiv:2207.05221)](https://arxiv.org/abs/2207.05221) - [GPT-4 Technical Report (arXiv:2303.08774)](https://arxiv.org/abs/2303.08774) - [Just Ask for Calibration (Tian et al., arXiv:2305.14975)](https://arxiv.org/abs/2305.14975) - [Taming Overconfidence in LLMs: Reward Calibration in RLHF (arXiv:2410.09724)](https://arxiv.org/abs/2410.09724) - [Why Language Models Hallucinate (arXiv:2509.04664)](https://arxiv.org/abs/2509.04664) - [Jev's Architecture Unmasked (Archer Hume)](https://archerhume.com/posts/jevs-architecture-unmasked/) - [TypeSafe Jev: calibration, decomposition, shadow eval (Rajesh Beri)](https://www.beri.net/article/typesafe-jev-typed-decision-model-calibration-decomposition-shadow-eval) - [Confidence (TypeSafe docs)](https://docs.typesafe.ai/confidence) - [A Gentle Introduction to Conformal Prediction (Angelopoulos and Bates, arXiv:2107.07511)](https://arxiv.org/abs/2107.07511) ## Related - [[Proper Scoring Rules]] - [[Reinforcement Learning for Calibrated Decisions (RLCD)]] - [[AI Post-Training]] - [[Reinforcement Learning From Human Feedback (RLHF)]] - [[AI Hallucination]] - [[AI Temperature]] - [[Confidence-Gated Routing]] - [[Self-Consistency]] - [[Jev]] - [[AI Evaluation]] - [[Conformal Prediction]] - [[Shadow Evaluation]]