# Relative vs Absolute AI Judgments When you ask an AI model to pick among options, there are two very different questions you might be asking: - **Relative**: "Which of these is the best fit?" The probabilities are spread across the options and sum to 1, so **something always wins**, even when every option is bad - **Absolute**: "Is *this* option a fit?" asked separately for each option. Each answer stands on its own, so **all of them can be low** That difference sounds academic until a system confidently suggests the wrong tool, cites the wrong passage or routes a ticket to the least-bad department. A relative judgment can't say "none of the above" unless you give it that option, and even then it's a weak signal. I think it's one of the most practical lessons in the [[TypeSafe AI]] docs, and it applies to any classifier, reranker or LLM you use to choose things. In [[Jev]]'s vocabulary ([[System One Primitives]]), a **Choice** is relative and a **Noul** (a yes/no probability) is absolute. ## Example 1: suggesting a skill to an agent The skill suggestion cookbook tries to help an agent pick the right skill among 182 (in 33 categories). A Choice over all 182 ranks them. But a ranking always has a top item, so on its own it would push a skill even for requests that need none. The fix is two steps. A Choice picks *which* skill; then a per-candidate "does this skill fit?" Noul decides *whether* to suggest anything at all (if the best Noul is under 0.30, suggest nothing). On a request to build a slide deck, the two judgments legitimately disagreed: the Choice picked `pptx-author` (the authoring skill, which is what the request needed), while the Nouls rated `powerpoint` (an editing skill) at 0.73 and `pptx-author` at 0.38. They answer different questions, so the system uses each for its own job. The check isn't magic either. A request to post to Mastodon still got the X skill suggested: there's no Mastodon skill, posting is an action, so the "fits" Noul landed at 0.56, above the 0.30 cut. The second step can only reject what the first step handed it. Results on 488 requests (315 with a matching skill, 173 without), with Claude Haiku 4.5 as the agent: | Condition | Wrong skill loaded | Needless load | |---|---|---| | Agent alone | 16.8% | 9.8% | | With the TypeSafe suggestion | **7.3%** | **4.0%** | | Oracle (handed the right answer) | 2.5% | 1.2% | And one line I keep thinking about: "A confident wrong suggestion is more persuasive than no suggestion." That's exactly why the "whether" check matters. ## Example 2: searching a document The semantic find cookbook splits GitHub's Terms of Service into 218 numbered lines and asks a Choice to point at the line that answers a query. Again, *some* line always ranks first. So an `exists` Noul rides along in the same request: does the document answer this at all? - "Who owns the code I upload?": exists 0.98, the right line at 0.95 - "Can they kick me off without warning?": exists 0.97, the right line at 0.97 - "Is there an arbitration clause?": the top line scored 0.86, but **exists was 0.14**. The document doesn't cover it - "Can minors use it with parental permission?": exists 0.46, i.e. partially covered (the age-13 rule ranked first but doesn't actually answer) The docs summarize it well: "The ranking tells you where to look; exists tells you whether it answers." ## Why the two don't agree (and shouldn't have to) Don't expect them to be consistent. The Jev 1.13 docs show that "Is the customer asking for a refund?" came back 0.22 as a Noul and 0.01 as a yes/no Choice on the same message. And "refund?" plus "something other than a refund?" summed to 1.19, not 1. They're different questions answered from different framings. So: - Never carry a threshold tuned on one form over to the other - Decide which question you actually mean, and ask it that way - If you need both "which" and "whether", ask both, in the same request (it's nearly free) ## The older names for this idea It's not new, just rarely applied: - **Closed-set vs open-set recognition.** A closed-set classifier assumes the true class is in its list; open-set recognition allows "none of these" - **Answerability.** SQuAD 2.0 (2018) added unanswerable questions to the classic reading-comprehension benchmark precisely because models trained on SQuAD 1.1 always extracted *some* span - **"Was it stated?" gates.** In the function calling cookbook, each optional argument gets a Noul asking whether the user mentioned it at all; if not, the default is used instead of confidently inventing a value (see [[Candidate-Then-Select Extraction]]) - **RAG filtering.** Retrieval returns the top k by similarity no matter what. A relevance check per passage can drop them all (see [[Retrieval-Augmented Generation (RAG)]] and [[Reranking]]) ## How to apply it - Any time your system picks among options, ask: *what happens when none of them is right?* - Add `none of the above` as a cheap first step, but for anything that matters, add an absolute check per candidate or an overall "does an answer exist?" question - Use the relative judgment for ranking and the absolute one for gating. Set the gate threshold from the cost of a wrong suggestion vs a missed one ## My take This is one of those ideas that looks obvious once named, yet I see it ignored everywhere: RAG pipelines that always stuff the top 5 passages into the prompt, tool routers that always pick a tool, classifiers without an "other" bucket. A model forced to choose will choose. If "nothing" is a valid answer, you have to ask for it explicitly. ## References - [Skill suggestion cookbook (TypeSafe docs)](https://docs.typesafe.ai/cookbooks/skill_suggestion) - [Semantic find cookbook (TypeSafe docs)](https://docs.typesafe.ai/cookbooks/semantic_find) - [Function calling cookbook (TypeSafe docs)](https://docs.typesafe.ai/cookbooks/function_calling) - [Jev 1.13 jaggedness (TypeSafe docs)](https://docs.typesafe.ai/model-jaggedness/jev-1.13) - [Know What You Don't Know: Unanswerable Questions for SQuAD (Rajpurkar, Jia, Liang, 2018)](https://arxiv.org/abs/1806.03822) ## Related - [[System One Primitives]] - [[Jev]] - [[Candidate-Then-Select Extraction]] - [[Reranking]] - [[Confidence-Gated Routing]] - [[Retrieval-Augmented Generation (RAG)]] - [[AI Agent Skills]] - [[Binary Classification]] - [[Data classification]]