# Self-Consistency "Self-consistency" means two different things in AI, and people mix them up constantly: 1. **A decoding technique** (Wang et al., 2022): ask a model the same question several times, let it reason differently each time, and take the majority answer 2. **A property of a model**: if you ask the same question about the same input ten times, do you get the same answer ten times? The first one uses inconsistency as a resource. The second one treats inconsistency as a bug. Both are worth understanding if you put AI inside software. ## 1. Self-consistency as a decoding trick Xuezhi Wang, Jason Wei, Denny Zhou and colleagues at Google published *Self-Consistency Improves Chain of Thought Reasoning in Language Models* in March 2022 (ICLR 2023). The idea: complex problems usually have several valid reasoning paths that lead to the same correct answer, while wrong paths tend to scatter. So instead of greedy decoding (one path), you sample many [[Chain-of-Thought (CoT) prompting|chain-of-thought]] paths and "marginalize out" the reasoning by voting on the final answers. The gains were large for such a simple trick: **+17.9% on GSM8K**, +11.0% on SVAMP, +12.2% on AQuA, +6.4% on StrategyQA and +3.9% on ARC-challenge. Two side effects matter in practice: - **The vote share is a confidence signal.** If 9 of 10 samples agree, you can trust the answer more than if it's 4 vs 3 vs 3. That's one of the most common black-box ways to estimate an LLM's uncertainty - **It costs N generations.** Ten samples means ten times the tokens and (without parallelism) ten times the latency [[Decision Models (DMs)|Decision models]] like [[Jev]] aim to give you that uncertainty signal in a single pass: the probability distribution *is* the vote, with no sampling needed. ## 2. Self-consistency as a property For a chatbot, a slightly different answer each time is fine, even charming. For a pipeline, it's a problem. If the same insurance claim gets "approve" on Monday and "review" on Tuesday, you can't test the system, audit it or explain it to a customer. And no, temperature 0 doesn't solve it: [[AI Temperature]] at zero reduces randomness, but batching, floating-point order and serving infrastructure still make LLM outputs move between runs. [[TypeSafe AI]] measured this in two cookbooks, with a clever setup: each call got a throwaway random `uid` field so caches couldn't return the previous answer, and every condition was repeated 15 times. **Yes/no questions (Nouls).** One auto insurance claim full of borderline calls (a loss at a race track, but in the parking lot while stationary; a rental line item with no rental coverage; a missing police report; an auto-triage note already saying "approved, pay full"), judged with a 14-question rubric: - Jev's mean per-question probability standard deviation was **0.0102**, the lowest of all probability-returning conditions - The LLMs (Claude Haiku 4.5, GPT-5.4 mini, and reasoning models GPT-5.5 and Claude Opus 4.8) moved between runs even at temperature 0, and disagreed with *themselves* on the judgment calls (exclusion, rental eligibility, fraud flag, manual review) - Even Jev wasn't perfectly stable where it mattered: `covered` ranged 0.43-0.53, crossing the 0.5 line, and `exclusion` 0.53-0.62 | Condition (per 14-question call) | Latency | Cost vs Jev | |---|---|---| | Jev | 111 ms | 1x ($0.000043) | | Claude Haiku 4.5, t=0 | 1.78 s | 42x | | GPT-5.5 reasoning | 11.1 s | 779x | | Claude Opus 4.8 reasoning | 13.9 s | 805x | **Multiple-choice questions (Choices).** One borderline moderation post (heated insults, an off-platform invite link, one prior strike, four reports), 8 questions: - Raw agreement on the top label: Jev 90.8%, LLMs 87.5-100%. Jev flipped on 2 of 8 questions (primary risk: harassment 11 times, violence 4 times) - Mean probability std: Jev 0.0098; Haiku at t=0 was actually lower (0.0012); other LLM conditions ranged 0.0245-0.0543 ## The fix: an "uncertain" band What turns a flaky decision into a stable one is usually not a better model but a better policy. Instead of one 0.5 cutoff, use three bands: - below 0.30 → no - 0.30 to 0.70 → uncertain, send to a human - above 0.70 → yes On the moderation post, a simple rule (top probability under 0.60 → uncertain) raised Jev's policy agreement from **90.8% to 99.2%**, with 25.8% of decisions going to review and **zero** cases where two runs produced two different concrete labels. No extra calls, just a threshold. This is [[Confidence-Gated Routing]] applied to stability. Note that single-pick LLM outputs (just a label, no probability) can't use this trick at all. ## Limits - **Repeatability isn't accuracy.** A model can be consistently wrong. TypeSafe's docs say so explicitly: these cookbooks measure stability, not correctness, and don't show superiority - **Bands have edges too.** Values near 0.30 or 0.70 can still flip; the band just moves the problem somewhere less frequent - **The setup can't separate causes.** Adding a random `uid` measures sensitivity to an irrelevant field *and* sampling noise at the same time - **Stability under rewording is a different test.** [[Diogo Almeida]] frames Jev's goal as "similar outputs for semantically similar inputs" (there's no seed parameter), which is broader than repeating the exact same input ## How to measure it on your own system 1. Pick 20-50 real inputs, including the borderline ones 2. Run each 10-15 times, with a random nonce in the input to defeat caching 3. Track the per-question probability std (or label agreement for label-only outputs) 4. Find the questions whose range crosses your threshold, and either reword them, add an uncertain band, or both ## My take I think self-consistency (the property) is underrated. Evals tell you how often a system is right; repeatability tells you whether you can trust *any single run* of it, which is what users experience. If a check flips between runs, I'd rather know before a customer does. And the cheapest fix is a policy change, not a model change. ## References - [Self-Consistency Improves Chain of Thought Reasoning in Language Models (Wang et al., arXiv:2203.11171)](https://arxiv.org/abs/2203.11171) - [Self-consistency with nouls cookbook (TypeSafe docs)](https://docs.typesafe.ai/cookbooks/consistency_noul_cookbook) - [Self-consistency with choices cookbook (TypeSafe docs)](https://docs.typesafe.ai/cookbooks/consistency_choice_cookbook) - [Jev: System One models for Prod, not God, with Diogo Almeida (Latent Space)](https://www.latent.space/p/jev) ## Related - [[Jev]] - [[Confidence-Gated Routing]] - [[AI Model Calibration]] - [[Chain-of-Thought (CoT) prompting]] - [[AI Temperature]] - [[AI Reasoning Models]] - [[AI Evaluation]] - [[Decision Models (DMs)]] - [[Machine Native Intelligence]]