# Reinforcement Learning with Verifiable Rewards (RLVR) RLVR is a way to [[AI Post-Training|post-train]] a language model where the reward comes from a check, not from a human. The model tries a maths problem, a program compares its final answer with the known result, and the model gets 1 if it's right and 0 if it's wrong. Same idea for code (do the unit tests pass?) or for instructions with hard constraints (is the answer really under 50 words?). That sounds almost too simple. But it's the recipe behind [[AI Reasoning Models]], and it's why models went from failing high school olympiad maths to acing it in about a year. ## Where the name comes from The term was coined in Tülu 3 (Lambert et al., Allen Institute for AI, November 2024, arXiv:2411.15124), which describes "a novel method we call Reinforcement Learning with Verifiable Rewards (RLVR)". The idea was already in the air, though: - **OpenAI o1** (announced 12 September 2024) used large-scale RL on chains of thought. On AIME 2024, GPT-4o solved 12% of problems on average; o1 solved 74% with one sample and 83% with a consensus of 64 samples - **GRPO** (Shao et al., DeepSeekMath, February 2024, arXiv:2402.03300) made this kind of RL cheaper. It drops PPO's separate value network and scores each sampled answer against the average of a group of answers to the same question - **DeepSeek-R1** (January 2025, arXiv:2501.12948) was the public proof. R1-Zero was trained with pure RL and rule-based rewards on top of DeepSeek-V3-Base, no human-written reasoning examples. Its AIME 2024 pass@1 went from 15.6% to 71.0% (86.7% with majority voting). See [[Deepseek]] ## How it works 1. Collect problems whose answers can be checked automatically (maths with a known result, code with tests, puzzles, formatted constraints) 2. Let the model generate a [[Chain-of-Thought (CoT) prompting|chain of thought]] and a final answer, several times per problem 3. Score each final answer with the verifier (usually just correct or incorrect) 4. Update the model to make the rewarded reasoning paths more likely (PPO, GRPO, or a variant) No learned reward model, no human rater in the loop. That removes a whole class of [[Reward Hacking]] against a fuzzy preference model. It doesn't remove all of it: models still find ways to game tests (hard-coding expected outputs, special-casing checks), because the verifier is itself a proxy ([[Goodhart's Law]]). ## Why it matters RLHF made models pleasant. RLVR made them *good at things you can check*. The difference is huge for anything with ground truth: maths, competitive programming, formal proofs, some agentic coding. And because rewards are cheap to compute, you can run a LOT of RL, which is where the long "thinking" traces and test-time compute scaling came from. ## Limits and criticism - **Only works where you can verify.** Most of real life (is this email urgent? is this contract risky?) has no unit test. RLVR helps less there - **Slow and expensive at inference.** Reasoning models produce thousands of thinking tokens before answering. [[TypeSafe AI]]'s launch post lists frontier model response times of "3 to 329 seconds" - **Jaggedness.** Almeida (TypeSafe's CEO) says RLVR "exacerbates jagged intelligence and doesn't integrate well with other software" (Latent Space, September 2026). The model becomes superhuman where rewards were dense and oddly weak next door ([[Jagged Intelligence]]) - **Binary rewards encourage guessing.** A 0/1 reward pays the same for a lucky guess and a sure answer, and gives nothing for saying "I'm not sure". Kalai, Nachum, Vempala and Zhang (OpenAI, arXiv:2509.04664, September 2025) argue this is exactly why models keep [[AI Hallucination|hallucinating]]: training and benchmarks reward good test-takers. Damani et al. (MIT, arXiv:2507.16806) showed you can fix part of it by adding a Brier score term to the reward (RLCR), which improved calibration without hurting accuracy. See [[Proper Scoring Rules]] That last point is the bridge to [[Reinforcement Learning for Calibrated Decisions (RLCD)|RLCD]]. RLVR asks "was the answer right?". RLCD asks "was the probability right?". The second question punishes confident guessing; the first one quietly rewards it. ## My take RLVR is one of the most important ideas in AI of the last two years; every "thinking" mode you've used since 2025 comes from it. But I think it's a hammer for a specific kind of nail. When the task is a quick, bounded judgment, making a model think for 30 seconds is waste. That's precisely the gap [[System One Models]] try to fill, and I find the split (reasoning models for System 2 work, decision models for System 1 work) useful. See [[System 1 and System 2 Thinking]]. ## References - [Tülu 3: Pushing Frontiers in Open Language Model Post-Training (arXiv:2411.15124)](https://arxiv.org/abs/2411.15124) - [Learning to reason with LLMs (OpenAI, o1)](https://openai.com/index/learning-to-reason-with-llms/) - [DeepSeekMath and GRPO (arXiv:2402.03300)](https://arxiv.org/abs/2402.03300) - [DeepSeek-R1 (arXiv:2501.12948)](https://arxiv.org/abs/2501.12948) - [Why Language Models Hallucinate (arXiv:2509.04664)](https://arxiv.org/abs/2509.04664) - [Beyond Binary Rewards: Training LMs to Reason About Their Uncertainty (RLCR, arXiv:2507.16806)](https://arxiv.org/abs/2507.16806) - [TypeSafe AI primer](https://docs.typesafe.ai/introduction/machine-learning-primer) - [Jev: System One models for Prod, not God, with Diogo Almeida (Latent Space)](https://www.latent.space/p/jev) ## Related - [[AI Post-Training]] - [[AI Reasoning Models]] - [[Reinforcement Learning From Human Feedback (RLHF)]] - [[Reinforcement Learning for Calibrated Decisions (RLCD)]] - [[Reinforcement Learning (RL)]] - [[Chain-of-Thought (CoT) prompting]] - [[Jagged Intelligence]] - [[Reward Hacking]] - [[Self-Consistency]] - [[Deepseek]]