# Benchmaxxing Benchmaxxing is raising a model's benchmark scores more than its actual capability. The word is slang (from "maxxing", maximizing), and on [[Hacker News]] it often shows up as a one-word verdict under a model launch: "benchmaxxed". It's [[Goodhart's Law]] applied to AI: once a handful of public benchmarks decide which model looks best, every lab has a reason to optimize for those benchmarks. It doesn't require bad intent. A lab can get there on purpose, by accident, or a bit of both. ## How it happens 1. **Contamination.** The test questions, or close variants, end up in the training data. Public benchmarks get copied to GitHub, blogs and forums, so a web-scale crawl picks them up even when nobody planned it 2. **Training on the tasks.** RL or fine-tuning directly on the benchmark tasks, or on near copies of them 3. **Learning the grader.** Many agentic benchmarks check requirements that the prompt never states. A model that has memorized those hidden rules passes more often without being better at the task. [[Reward Hacking]] is the close cousin: the model learns what the grader rewards, not what the task asks 4. **Burning tokens.** Over-saturating reasoning buys points on hard benchmarks, and you pay for it in latency and cost on daily work (the main complaint about [[Kimi K3]]) 5. **Picking the comparison.** Not a training problem, but the same effect on the reader: choose the benchmark version, the harness and the rivals that make your model look best ## The Kimi K3.1 audit (November 2026) The clearest public case I know is R40 Research's audit of [[Kimi K3.1 Pro]] (Joachim Schaeffer and Tilman Räuker, 3 November 2026). K3.1 topped the [[Artificial Analysis]] Terminal-Bench 4.0 leaderboard at 67.2%. Moonshot's API returns the raw reasoning, so they could read it on every attempt. - **It cites instructions it never got.** In 23.3% of its 189 Terminal-Bench attempts, K3.1's reasoning mentions "instructions" that aren't in the prompt but that the hidden tests check. Kimi K3 never does this; GLM-5.3 did it once. Sometimes K3.1 even guesses where they come from: "Instructions likely distilled pitfalls from previous agents" - **Ask, and it lists them.** Prefill its reasoning with a request to list its instructions, and it names hidden requirements 64.6% of the time (K3: 17.6%) - **The lead sits in those tasks.** On the 16 tasks where it states a hidden rule before seeing any evidence, it passes 70.8% of attempts, while five Anthropic and OpenAI frontier models average 34.0%. On the other tasks it's in the same range as everyone else. Removing the attempts that exploit known grader loopholes also drops its score from 66.7% to 61.4% - **It ignores the task to obey the rule.** This is the test I like most. They edited one file per task so that it contradicts a hidden rule. K3.1 ignored the edit and failed on three of four tasks, while K3, GLM-5.3 and [[GPT-6 Astra]] followed it. In one case its reasoning says "Should obey even if comments contradictory". As a control, it does follow edits that don't conflict with a memorized rule - **It knows where it is.** On HLE (Humanity's Last Exam), its reasoning mentions "previous tasks" or "this platform" on 24.9% of the questions (K3: 3.2%, six Anthropic and OpenAI models: at most 3.9%), and it names HLE itself on 8.4% Their hypothesis is training on the benchmark tasks with a teacher that saw the hidden grading rules (e.g., on-policy self-distillation, see [[Knowledge Distillation]]). The authors are careful about what they can claim: it's a rapid study, not peer reviewed, built on LLM judges with human spot checks, and observed behavior "cannot prove how models were trained", nor whether it was intentional. The Hacker News pushback is worth keeping too. Human-preference arenas like Design Arena are much harder to game, and K3 sat at the top of its open-weight ranking for a long time. One commenter who tested K3.1 drew the line I find most useful: "obviously benchmaxxed compared to the western closed models, but it is not compared to the OSS competitors." Both can be true at once. ## How to spot it - **Read the reasoning, when you can.** A model that recalls hidden rules or names the benchmark gives itself away - **Perturb the task.** Change a detail so the memorized answer becomes wrong. A capable model follows the new task; a memorizing one sticks to the old answer - **Check which tasks it solves.** Models with similar ability tend to fail on similar tasks. The tasks K3.1 solved barely matched the ones the frontier models solved (Spearman ρ 0.20, against at least 0.55 among the frontier models) - **Compare with less-targeted tests.** In October 2026, [[StarSkirmish Bench]] scores barely tracked the Artificial Analysis Intelligence Index (ρ 0.26) but closely tracked less famous coding tests - **Prefer held-out and blind signals.** Private test sets, preference arenas, blind trials. [[MiniMax M3.1 Flash]] looked near-frontier on its own charts and got a lukewarm verdict from developers who had used it blind as a stealth model - **Run your own tasks.** Your work is the only benchmark nobody trained on (cfr [[AI Evaluation]]) ## My take I don't read a launch table as evidence anymore. I read it as the lab's claim, and I wait for three things: independent runs, a blind or human-preference signal, and my own tasks. The K3.1 audit shows why. A model can top a leaderboard while most of its lead sits in the tasks where it "remembers" the grader. That said, "benchmaxxed" has become a reflex comment, often posted before anyone ran the model. An accusation isn't a measurement either. Read the trace, contradict the rule, compare the solved tasks: that's the kind of evidence I want before writing a model off. ## References - Joachim Schaeffer and Tilman Räuker (R40 Research), "Kimi K3.1 Pro Recalls and Enforces Hidden Test Rules" (2026-11-03): https://r40.org/papers/kimi_k31_benchmark_audit.pdf - Hacker News, "Kimi K3.1 Pro model weights are out" (audit and Design Arena discussion): https://news.ycombinator.com/item?id=50366999 ## Related - [[Goodhart's Law]] - [[Reward Hacking]] - [[AI Evaluation]] - [[Knowledge Distillation]] - [[Kimi K3.1 Pro]] - [[Kimi K3]] - [[MiniMax M3.1 Flash]] - [[StarSkirmish Bench]] - [[Artificial Analysis]] - [[SWE-Bench]]