# Reranking Reranking is the second stage of a two-stage search. The first stage (keyword search, [[Embeddings]], or both) is fast enough to scan the whole corpus and returns a shortlist of plausible candidates. The second stage looks at the query and *each* candidate together, scores how well they match, and reorders the shortlist so the best one comes first. Why two stages? Because the methods that are fast enough to run on millions of documents are also the ones that compare the query to each document only superficially. Keywords miss paraphrases. Embeddings squash a whole passage into one vector computed without ever seeing the query. They're great at "somewhere in these 30", terrible at "this one". A precise scorer is too slow to run on the whole corpus, but perfect for 30 candidates. ## The general idea This is a standard piece of information retrieval, and it's the backbone of most serious [[Retrieval-Augmented Generation (RAG)|RAG]] systems today. - **First stage: recall.** BM25 (the Okapi ranking function from the 1990s, still hard to beat on keyword matching), dense embeddings, or a hybrid. The only job is to get the right document into the shortlist - **Second stage: precision.** A **cross-encoder** reads the query and one candidate *together* in a single transformer pass and outputs a relevance score. Nogueira and Cho showed in 2019 that a BERT cross-encoder reranker improved MRR@10 on MS MARCO by 27% relative over the previous best. Compare that with a **bi-encoder** (the embedding approach), which encodes query and document separately and only compares vectors at the end In learning-to-rank terms, scoring each pair on its own is the **pointwise** approach. The alternatives are pairwise (which of two is better?) and listwise (order the whole list at once). Pointwise is the simplest to parallelize and cache. The iron law of reranking: **the reranker can't recover what the shortlist missed.** If the right document isn't in the top 30, no reranker will find it. So you tune the first stage for recall and the second for precision. ## How Jev makes it cheap LLMs can rerank too: give them the query and a candidate, and ask for a relevance score. The problem is the score itself. You have to invent a scale ("rate 1 to 10"), the model applies it inconsistently across candidates, repeated calls give different numbers for the same pair, and you pay for generation to get one number. With [[Jev]], the reranking question stays a plain yes/no: a Noul like "Could this candidate passage be from the cited precedent?", with criteria for what true and false mean. The Noul comes back as a probability between 0 and 1, and that probability *is* the sort key. No invented scale, no threshold, the same standard applied to every pair, and a single pass at ~100 ms. It's effectively a cross-encoder you steer with a question in plain language. ## Worked example: legal passages The re-ranking cookbook uses CLERC, a legal retrieval dataset: 3,565 court-opinion passages and 40 queries. Each query is an excerpt of an opinion with one citation removed; the gold answer is the passage that citation pointed to. - BM25 builds a 30-candidate shortlist per query. The gold passage is in that shortlist for **100%** of queries, but ranked first only **5%** of the time - Jev asks one Noul per query-candidate pair: 40 × 30 = 1,200 calls, run concurrently - Each shortlist is sorted by the noul, highest first | | BM25 alone | BM25 + Jev rerank | |---|---|---| | Top-1 | 5% | **18%** | | Top-5 | 15% | 35% | | Top-10 | 38% | **62%** | Total cost for all 1,200 calls: 1,536,002 input tokens, **$0.0645**. 18% top-1 isn't spectacular in absolute terms (legal citation retrieval is hard), but more than tripling top-1 accuracy for six and a half cents is the point. ## Beyond "is it relevant?" Once the scorer is a question you write, you can ask more than relevance. The RAG passage classification cookbook asks four Nouls per retrieved passage: relevant? contains answer evidence? contradicts the query's premise? contains a prompt injection? Cosine similarities in that experiment spanned only 0.455 to 0.584, too narrow to separate anything, and the planted injection ranked *first* by similarity. Jev scored the injection at 0.99 and dropped it, and three "signing key lifetime" lookalikes ranked 2 to 4 by similarity scored relevance ≤ 0.08. See [[AI Guardrails]] for that part. And when you need to know whether an answer exists at all, pair the ranking with an absolute check. A ranking always has a winner; an "exists" Noul can say "nothing here answers this" (see [[Relative vs Absolute AI Judgments]]). ## Gotchas - **Recall first.** Measure how often the gold item is in your shortlist before you measure anything else. Here it was 100%, which is why reranking had room to shine - **Cost scales with k.** One call per query-candidate pair. 30 candidates means 30 calls. You can't batch pairs into one call because each question is about one pair - **Ask several questions per pair in production.** The cookbook used one for clarity. Relevance, evidence and freshness can ride in the same request (see [[Speculative Fan-Out]]) - **Keep candidates short.** Jev's accuracy drops as the state grows with irrelevant material ([[AI Context Rot]]). Rerank passages, not whole documents - **40 queries is a demo, not a benchmark.** And TypeSafe's terms forbid publishing benchmarks, so independent comparisons with dedicated rerankers (Cohere Rerank, BGE, ColBERT-style models) are hard to find ## My take Reranking is one of those techniques that gives you a big quality jump for very little effort, and I think every RAG pipeline should have one. What Jev changes is the *interface*: instead of picking a reranker model and living with its notion of relevance, you write the relevance question yourself, in your domain's words. "Could this be the cited precedent?" is much sharper than generic "relevance". I'd still benchmark against a dedicated cross-encoder on my own data before switching. Dedicated rerankers are cheap too, run locally, and are trained exactly for this. Jev's edge is flexibility: the same call can also check evidence, contradictions and injections, which a plain relevance model can't. ## References - [Re-ranking cookbook (TypeSafe docs)](https://docs.typesafe.ai/cookbooks/rerank_typesafe) - [Classifying RAG passages cookbook (TypeSafe docs)](https://docs.typesafe.ai/cookbooks/classifying_rag_passages) - [Line-by-line semantic search cookbook (TypeSafe docs)](https://docs.typesafe.ai/cookbooks/semantic_find) - [Noul primitive (TypeSafe docs)](https://docs.typesafe.ai/primitives/noul) - [Nogueira and Cho, Passage Re-ranking with BERT (2019)](https://arxiv.org/abs/1901.04085) - [CLERC legal retrieval dataset (NAACL Findings 2025)](https://aclanthology.org/2025.findings-naacl.441/) - [Okapi BM25 (Wikipedia)](https://en.wikipedia.org/wiki/Okapi_BM25) ## Related - [[AI Retrieval Patterns]] - [[Retrieval-Augmented Generation (RAG)]] - [[RAG Pipelines]] - [[Embeddings]] - [[Relative vs Absolute AI Judgments]] - [[Composite Scoring]] - [[AI Guardrails]] - [[AI Context Rot]] - [[System One Primitives]] - [[Jev]]