# Ori Eval Ori Eval is [[OpenRouter]]'s tool for writing your first eval. Their pitch, and it's the right one: > There's no definitive best model, only the best model for each task. An eval is a `*.eval.ts` file. It runs your agent against prompts you choose, checks which tools got called, and uses an LLM judge to score the open-ended answers. Because it routes through OpenRouter, the same eval runs against models from every major lab without rewriting anything. ## Getting it ```sh curl -fsSL https://openrouter.ai/labs/ori/install.sh | bash ori login ``` Needs [[Bun]] to run evals. There's also a spawn endpoint aimed at agents rather than humans, which hands your coding agent the full onboarding guide in one command: ```sh curl -fsSL https://openrouter.ai/skills/spawn-ori-eval ``` Worth pausing on that: the onboarding path for the tool is *have your agent read this and set it up for you*. Documentation as a skill, delivered by curl. ## The design decision that matters **Runs lock both the harness and the model.** That single choice is what separates an eval from a vibe check. If the model version and the surrounding harness float between runs, a score change tells you nothing: you can't tell whether your prompt improved, the harness changed, or the provider silently shipped a new checkpoint. Pin both, and a score delta means *your agent changed*, which is the only question worth asking. Three things get measured: whether the right tools were called, whether open-ended output holds up under an LLM judge, and how all of that shifts across models. ## Why I keep this note **I have no evals, and I should.** I maintain a large library of skills, agents and panels, all of which are prompts doing real work, and every change to them is currently validated by reading the output and forming an impression. That's fine for one change and useless across a hundred. The failure mode is silent: a skill degrades slightly, I don't notice, and the vault fills with worse output. **Per-task model selection is a live question here.** My setup already tiers models by task, and that tiering is based on intuition and reputation rather than measurement. "Which model is actually best for *my* newsletter drafting, *my* note-writing, *my* style checks" is answerable with an eval and not otherwise. The [[Kimi K3]] field report makes the same point from the other direction: the useful finding there was a specific build that failed, discovered by a check, not by reading the diff. **Pinning the harness generalises past this tool.** Any time you compare two prompt versions, anything unpinned underneath makes the comparison meaningless. That's the same lesson as the [[Qwen 3.8]] benchmark table, where Terminal Bench numbers were measured through different harnesses and are therefore not comparable. Same error, one at lab scale and one at mine. The obvious first target: take the two or three skills I lean on daily, write one eval each, and find out whether the cheaper model was fine all along. ## References - [Ori Eval on OpenRouter](https://openrouter.ai/ori/eval) - [OpenRouter's announcement (2026-08-03)](https://x.com/OpenRouter/status/2084301100078027143) - Install: `curl -fsSL https://openrouter.ai/labs/ori/install.sh | bash` - Agent onboarding: `curl -fsSL https://openrouter.ai/skills/spawn-ori-eval` ## Related - [[OpenRouter]] - [[AI Agents]] - [[AI Agent Skills]] - [[Bun]] - [[Kimi K3]] - [[Qwen 3.8]] - [[Claude Code]]