# Zero-Shot Classification Zero-shot classification means sorting text into categories the model has never seen a single training example of. You hand it the text and a list of labels ("billing", "shipping", "account access"), and it picks one (or scores all of them), with zero labeled examples. Hence "zero-shot". Why does this matter? Because the classic way to build a classifier is painful: collect thousands of labeled examples, train, evaluate, retrain whenever the categories change. Zero-shot turns the label list into an input. Add a category and you're done. That's a massive shortcut for prototypes, long-tail categories and anything that changes every week. ## The NLI trick (2019) The approach that made zero-shot classification practical comes from Wenpeng Yin, Jamaal Hay and Dan Roth (*Benchmarking Zero-shot Text Classification*, EMNLP 2019, arXiv:1909.00161). The idea is clever: reuse a model trained for **natural language inference** (NLI), the task of deciding whether a *premise* entails a *hypothesis* (see [[Natural Language Processing (NLP)]]). - The text to classify becomes the premise: "My card was charged twice for the same order" - Each label becomes a hypothesis through a template: "This example is about billing.", "This example is about shipping." - The NLI model scores how strongly the premise *entails* each hypothesis - The label with the highest entailment probability wins The model never learned about billing tickets. It learned what "entails" means, and that's general enough to cover almost any label you can phrase as a sentence. Yin et al. also pushed for a harder, more realistic setup they call "label-fully-unseen" (no training data for the task at all), and tested it well beyond topic labels: emotions like joy or anger, and situations like "medical assistance" or "water shortage". This became Hugging Face's `zero-shot-classification` pipeline, usually with `facebook/bart-large-mnli` (BART fine-tuned on the MultiNLI dataset): ```python from transformers import pipeline classifier = pipeline("zero-shot-classification", model="facebook/bart-large-mnli") classifier( "My card was charged twice for the same order", candidate_labels=["billing", "shipping", "account access"], ) ``` The pipeline's default template is "This example is {}.", and you can pass your own with `hypothesis_template`. By default, the entailment scores are normalized across labels (exactly one label is right). With `multi_label=True`, each label gets its own independent yes/no probability. ## Other ways to do it - **Embedding similarity.** Embed the text and a short description of each label with the same model ([[Embeddings]]), pick the label with the highest cosine similarity. Cheap and fast, but similarity isn't the same as "belongs to this category" (a complaint about billing and praise of billing look alike) - **Prompting an LLM.** Ask a [[Large Language Models (LLMs)|large language model]] "Which category fits this ticket: A, B or C?" and parse the answer. Flexible and smart, but slow, expensive, and the output is text you have to parse (see [[LLM Structured Outputs]]) - **Reading option logits.** Instead of letting the LLM write, read its probability for each option letter in a single forward pass. That's how MMLU is scored (Hendrycks et al., arXiv:2009.03300), and it gives you a probability distribution without any generation ## Known weaknesses - **Label wording.** "Billing" vs "billing issue" vs "payment problem" can shift results a lot. The label *is* the prompt, so you end up doing prompt engineering on your category names - **Option order and letter bias.** Zheng et al. (*Large Language Models Are Not Robust Multiple Choice Selectors*, ICLR 2024, arXiv:2309.03882) tested 20 LLMs and found they favor certain option letters (A, B, C, D) regardless of content. Move the right answer to a different position and accuracy changes. Their fix, PriDe, estimates that prior bias and removes it at inference time - **Probabilities aren't calibrated.** An entailment score of 0.9 doesn't mean "right 90% of the time". Nobody trained it to (see [[AI Model Calibration]]) - **One text, one pass per label.** The NLI approach runs the model once per (text, label) pair. Fine for 5 labels, painful for 200 ## "Jev isn't new" This lineage is what the critics point to. KDnuggets (Abid Ali Awan) put it plainly: "Classification is not new. Intent detection is not new. Zero-shot classification is not new." NLI-based zero-shot classification got popular around 2019-2020. And they're right that the core mechanism is familiar. A [[System One Primitives|Noul]] ("is this statement true about the state?") is essentially an entailment probability. A Choice over options is essentially reading option logits. But KDnuggets also concedes that "the problem is old; the architecture and product around it may be new." That matches what I see. What [[Jev]] adds over a 2019-style zero-shot pipeline: 1. **Generality.** One model for yes/no, multiple choice (up to 255 options), and rubric scores, on any domain, with instructions and criteria instead of a fixed template 2. **Calibration as a training goal.** [[Reinforcement Learning for Calibrated Decisions (RLCD)|RLCD]] explicitly targets probabilities that match frequencies. bart-large-mnli never had that goal (independent tests put Jev's calibration error at 0.031 on MMLU and 0.107 on support tickets, so the goal isn't always met) 3. **Many questions over shared state.** You encode a document once and ask it ten questions in parallel. The NLI approach re-reads the text for every label 4. **Options read jointly.** Archer Hume found that adding an irrelevant option shifts the odds between the others, so Jev compares options instead of scoring each one in isolation The open alternatives ([[SemIf]], [[Laya]]) show how thin the line is: they rebuilt the interface on top of open models within days. The moat, if there is one, is in the training data and calibration, not in the idea. ## My take I think the "it's not new" critique is correct and mostly irrelevant. Zero-shot classification has worked since 2019, but it was fiddly, uncalibrated and slow once you had many labels. What changes adoption is packaging: typed outputs, probabilities you can threshold, cheap enough to call thousands of times. If you already run a bart-large-mnli pipeline that works, keep it. If you've been asking an LLM to write JSON to pick a category, you've been doing zero-shot classification the expensive way. ## References - [Benchmarking Zero-shot Text Classification (Yin, Hay and Roth, EMNLP 2019, arXiv:1909.00161)](https://arxiv.org/abs/1909.00161) - [facebook/bart-large-mnli (Hugging Face)](https://huggingface.co/facebook/bart-large-mnli) - [Measuring Massive Multitask Language Understanding (Hendrycks et al., arXiv:2009.03300)](https://arxiv.org/abs/2009.03300) - [Large Language Models Are Not Robust Multiple Choice Selectors (Zheng et al., arXiv:2309.03882)](https://arxiv.org/abs/2309.03882) - [What everyone is getting wrong about TypeSafe AI's Jev (KDnuggets)](https://www.kdnuggets.com/what-everyone-is-getting-wrong-about-typesafe-ais-jev) - [Jev's Architecture Unmasked (Archer Hume)](https://archerhume.com/posts/jevs-architecture-unmasked/) ## Related - [[Decision Models (DMs)]] - [[Binary Classification]] - [[Jev]] - [[System One Primitives]] - [[SemIf]] - [[Laya]] - [[Natural Language Processing (NLP)]] - [[Embeddings]] - [[LLM Structured Outputs]] - [[AI Model Calibration]] - [[Relative vs Absolute AI Judgments]]