# Proper Scoring Rules A scoring rule grades a probabilistic forecast once you know what happened. You said 70% chance of rain; it rained; how good was your forecast? A scoring rule is **proper** when your best expected score comes from reporting what you actually believe. It's **strictly proper** when honesty is the ONLY way to get the best expected score: any hedging or exaggeration costs you, on average. That's a big deal. It means the grading system itself makes lying pointless. You can't game a strictly proper rule by sounding more confident or playing it safe. Which is why I think this old idea from weather forecasting is the cleanest way to understand what "calibrated AI" should mean. ## The two classics - **Brier score** (Glenn Brier, *Monthly Weather Review*, 1950). The mean squared error between your probability and what happened (1 if it happened, 0 if not). Say 0.9 and it happens: (0.9 - 1)² = 0.01. Say 0.9 and it doesn't: (0.9 - 0)² = 0.81. Lower is better. Bounded, so one terrible forecast can't ruin you - **Log score / log loss** (I. J. Good, 1952). The negative log of the probability you gave to what actually happened. Say 0.01 and it happens: -log(0.01) ≈ 4.6. Say 0 and it happens: infinite penalty. It punishes confident mistakes brutally. It's the same thing as cross-entropy, the loss used to pretrain every [[Large Language Models (LLMs)|LLM]] Allan Murphy showed in 1973 that the Brier score splits into three parts: **reliability** (are your numbers calibrated?), **resolution** (do you separate events that happen from events that don't?) and **uncertainty** (how hard the problem is to begin with). So a proper score rewards both honesty and usefulness. Always saying the base rate is honest but has zero resolution. ## What is NOT proper - **Accuracy** (right or wrong, 0/1). It only looks at the top answer, so there's no reward for saying 0.55 instead of 0.99. It pays a confident guess and a careful one the same - **Linear score** (the probability you gave to the outcome). Sounds reasonable, but the best strategy is to put 100% on whatever you think is most likely. It rewards exaggeration This is why most AI benchmarks quietly push models toward overconfidence. Kalai et al. (OpenAI, arXiv:2509.04664, 2025) make exactly this argument: evaluations scored on right/wrong reward guessing over "I don't know", so we train good test-takers and get [[AI Hallucination|hallucinations]]. ## Why it matters for AI This is the thread that ties the Jev story together: 1. **Pretraining is proper.** Next-token cross-entropy is a log score. That's why base models come out well calibrated (see [[AI Model Calibration]]) 2. **[[Reinforcement Learning From Human Feedback (RLHF)|RLHF]] is not.** A reward model trained on human preferences is a proxy that likes confident-sounding answers, and optimizing it erodes calibration (the [[GPT4|GPT-4]] report measured ECE going from 0.007 to 0.074 after PPO). That's [[Goodhart's Law]] applied to probabilities 3. **[[Reinforcement Learning with Verifiable Rewards (RLVR)|RLVR]] is not either.** A 0/1 correctness reward is accuracy, which is improper 4. **Adding a proper term fixes a lot.** RLCR (Damani et al., MIT, arXiv:2507.16806) rewards correctness plus a Brier term and proves the combination gives accurate AND calibrated models. Rewarding Doubt (Bani-Harouni et al., arXiv:2503.02623) uses the log scoring rule as an RL reward for verbalized confidence 5. **[[Reinforcement Learning for Calibrated Decisions (RLCD)|RLCD]]** probably sits in this family. TypeSafe hasn't published its reward function, but a proper scoring rule over [[Jev]]'s typed outputs is the most plausible reading of "epistemically honest probabilities" [SPECULATIVE] A note of caution: a strictly proper rule can't be gamed by misreporting your confidence, but it can still be gamed through the data. Choose easy training questions and you'll look calibrated on easy questions. Calibration is always relative to the distribution you scored on. ## Beyond AI The same math runs forecasting tournaments. Philip Tetlock's Good Judgment Project scored forecasters with Brier scores, and prediction markets pay out in a way that's closely related. If you've ever wondered how to grade a forecast of a one-off event ("will this project ship by March?"), the answer is: you can't grade one, but you can grade a track record with a proper score. The formal reference is Gneiting and Raftery, *Strictly Proper Scoring Rules, Prediction, and Estimation* (Journal of the American Statistical Association, 2007). ## My take I love this concept because it answers a question that sounds philosophical ("how do you reward honesty?") with a formula. If you want an AI (or a team, or yourself) to tell you how sure it really is, grade it with a proper score and stop rewarding bold, lucky calls. I'd even use it on my own predictions: write down a probability, check later, compute a Brier score. I suspect most of us would find the result humbling. ## References - [Scoring rule (Wikipedia)](https://en.wikipedia.org/wiki/Scoring_rule) - [Strictly Proper Scoring Rules, Prediction, and Estimation (Gneiting and Raftery, 2007)](https://doi.org/10.1198/016214506000001437) - [Brier score (Wikipedia)](https://en.wikipedia.org/wiki/Brier_score) - [Why Language Models Hallucinate (arXiv:2509.04664)](https://arxiv.org/abs/2509.04664) - [RLCR: Beyond Binary Rewards (arXiv:2507.16806)](https://arxiv.org/abs/2507.16806) - [Rewarding Doubt (arXiv:2503.02623)](https://arxiv.org/abs/2503.02623) - [GPT-4 Technical Report (arXiv:2303.08774)](https://arxiv.org/abs/2303.08774) ## Related - [[AI Model Calibration]] - [[Reinforcement Learning for Calibrated Decisions (RLCD)]] - [[AI Post-Training]] - [[Goodhart's Law]] - [[AI Hallucination]] - [[Reward Hacking]] - [[Jev]]