# Claude Code Plugin Evals `claude plugin eval` runs a [[Claude Code Plugins|plugin]] (or a [[Claude Code Skills|skill]] shipped in one) against a suite of test cases, scores each run, and by default **runs every case again without the plugin** so you can see what the plugin actually contributes. Shipped in v2.1.269 (week of 2026-09-07). The question it answers is the one every skill author should be asking and almost nobody measures: does this skill change what Claude does, or would Claude have done it anyway? ## How a run works - An eval suite lives in `evals/` inside the plugin. Each case is a directory with a `prompt.md` (what a user would type, phrased naturally, not naming the skill) and one or more graders under `graders/` - For each run, Claude Code starts a fresh, isolated, non-interactive session with only your plugin loaded, sends the prompt, and lets Claude work until it finishes or hits the case's turn or time limit - Each case runs **three times by default**, because one run of a non-deterministic agent tells you little. A run's score is the fraction of graders that passed (weighted if you set weights); the case's score is the mean across runs. A case passes at `--threshold`, `1.0` by default - With the plugin under test, each case runs in two arms: **with** and **without**. You get `WITH`, `W/OUT`, and `Δ`. A case that scores 1.0 in both arms passed for reasons that have nothing to do with your plugin ``` CASE WITH W/OUT Δ RUNS COST NOTES ... Report: evals/results/<timestamp>/report.html ``` ## Graders Six types. Four are free (computed from the transcript and files); two call a judge model and cost money. | Type | Cost | Checks | |---|---|---| | `regex` | free | a pattern over the reply or a file | | `tool_used` | free | a tool (or skill) was invoked; `input_match` can pin the arguments | | `tool_order` | free | tools were called in a given order | | `file_exists` | free | a file was created | | `llm` | judge call | a rubric written as concrete PASS / FAIL conditions | | `baseline` | judge call | compares against the no-plugin arm | No custom-code graders. The judge is a small fast model by default; pass `--judge-model sonnet` for nuanced rubrics. A `tool_used: Skill` grader can never pass in the without-arm, so it's excluded from scoring in both arms and reported as an indicator only. Same for anything you mark `arm: with-only`. Set `arm: both` for a "must **not** invoke the skill" check. Advice from the docs that matches my own experience with skill health checks: give each case one grader on the result and one on how Claude got there. Together they tell you whether the answer was right *and* whether your plugin produced it. For long outputs, grade a file with `regex`; keep `llm` graders for short replies. ## Getting started ```bash cd my-plugin claude plugin eval init # describe what a good result looks like; Claude interviews you, proposes cases + graders, pilots them, writes the files claude plugin eval . # runs every case, with and without the plugin claude plugin eval . --case <name> --runs 1 --ablation none # cheap iteration on one case ``` The most common first finding, per the docs: `Δ` near zero with the `tool_used: Skill` grader failing. Claude isn't picking your skill on natural phrasing. Fix the skill's `description`, re-run, compare. That's the whole loop. ## Useful options | Option | Default | What it does | |---|---|---| | `--runs <n>` | 3 | runs per case per arm | | `--ablation none` | `with-without` | skip the no-plugin arm; halves cost | | `--threshold <0..1>` | 1.0 | pass mark; any case below it exits 1 | | `--max-cost-usd <usd>` | none | ceiling on the list-price estimate; exits 2 with partial results when hit | | `--judge-model <model>` | small fast model | model for `llm` and `baseline` graders | | `--eval-dir <path>` | `evals/` | alternate suite location (also settable in `plugin.json` under `experimental.evals`) | | `--json` | | machine-readable result, one object on the last line | | `--trust-plugin` | | skip the first-run trust prompt (CI) | Exit codes: 0 all cases at or above threshold; 1 a case failed or a file didn't load; 2 partial run (cost ceiling hit or credential rejected). ## Mocks Skills that call [[Model Context Protocol (MCP)|MCP]] tools can be evaluated without the real service. One Markdown file per tool under `evals/mocks/<server>/<tool>.md` (suite-wide) or a case's own `mocks/`. The body is what the tool returns; `{{input.<field>}}` interpolates the call's input; an `expect:` block aborts the run with score 0 if Claude sends the wrong input, which turns the mock into an assertion. `type: agent` lets a small model answer as the server, and answers get recorded under `mock-recordings/` for replay. ## Cost Every run and every judge grader is a real model call against your plan or API bill. A suite makes roughly cases × runs agent runs with the plugin, as many again for the baseline, plus three judge calls per `llm` grader per run. For every-change CI suites: free graders only, `--ablation none`, pinned models, a `--max-cost-usd` ceiling, `--threshold 0.8` or so. ## Running it against a skills folder (not a plugin) My skills live in `.claude/skills/`, not in a plugin, and a bare skills folder does not resolve as a plugin (no baseline arm, `plugins: []`). What works: a throwaway wrapper directory with a `.claude-plugin/plugin.json` manifest and a `skills` symlink to the real folder; the runner loads all 400+ skills into each child session, which is the realistic condition for "does Claude pick the right one". What I learned doing it on 2026-09-22: - `init` is an interactive interview; for hundreds of skills, generate cases from each skill's trigger vocabulary instead. One case per skill, prompt = its first trigger phrase verbatim, one free `tool_used: Skill` grader with `input_match` on the skill name, `allowed_tools: [Read, Glob, Grep, Skill]`, `max_turns: 2`. No judge, no writes, nothing touches the vault: each run gets a throwaway home and cwd. - The skill fires on turn 1 or not at all. `max_turns: 2` is enough for a selection test; turns 3+ are the skill body executing against an empty workspace, pure cost. - For skills whose trigger phrases collide, add a `tool_used` grader with `min: 0` / `max: 0` per competing skill (weight 1, the fire grader weight 10). The report then names who won without a judge. - With ~420 skills loaded a run costs about $0.40 (opus, list price) and 15 s. That is the real per-skill price of the test; the skill listing dominates the context. - Runs bill whichever auth wins: an `ANTHROPIC_API_KEY` in the environment takes precedence over the claude.ai login. Mine ran dry mid-suite and the runner kept going, logging 94 zero-cost "Credit balance is too low" runs that look like "never fires" unless the analysis separates errored runs from real ones. Unset the key to run on the plan. - First slice, 36 colliding skills × 3 runs: 26 fire every time, 5 flaky, 5 never; and in no failed run did a competitor fire. The collisions were not the problem. Claude answering *without any skill* was. Tooling and per-skill results: `.claude/skills-eval/` in the vault (`README.md`, `LATEST.md`). ## What I take from it This is the missing half of `/skill-doctor`. That command tells you which skills cost context and never fire; this one tells you whether the skills that *do* fire are earning their keep. Between them you can finally run a skill library like a codebase: measure, prune, regress-test on every model release. For a vault with several hundred skills, "does Claude pick this skill on natural phrasing" is the test I'd write first, everywhere. ## References - Documentation: https://code.claude.com/docs/en/plugin-evals - Announcement (ClaudeDevs, 2026-09-11): https://x.com/ClaudeDevs/status/2098500999656923145 - Week 37 digest: https://code.claude.com/docs/en/whats-new/2026-w37 - Addy Osmani (2026-09-21): https://x.com/addyosmani/status/2101921871092658655 ## Related - [[Claude Code]] - [[Claude Code Plugins]] - [[Claude Code Skills]] - [[Claude Code Hooks]] - [[Ori Eval]] - [[AI Agent Skills]]