# Candidate-Then-Select Extraction Candidate-then-select is a way to extract data from text where the model never writes the value. Code proposes the possible answers first (every email address a regex can find, every allowed value of an enum, every month of the year). The model only *picks* among them. Then code copies the picked candidate verbatim and does any formatting or math. Why bother? Because the classic failure of LLM extraction is a value that looks right and isn't: a transposed digit in an invoice amount, a phone number stitched from two numbers, a date that's one day off, a field "filled in" with something the document never said. When the model can only choose from spans that literally exist in the input, a whole class of those errors disappears. It can still pick the wrong candidate. It can't invent one. ## The general idea The idea has solid roots: - **Extractive question answering.** SQuAD-style readers (2016) don't generate answers; they point at a start and an end position inside the passage - **Pointer networks** (Vinyals, Fortunato, Jaitly, 2015) output positions in the input instead of words from a vocabulary - **Slot filling** in classic dialog systems fills each slot from a closed set of values - **Constrained decoding** forces an LLM to only emit tokens that match a grammar or a list The pattern has two halves, and they have different jobs. **Candidate generation must have high recall** (over-find, never miss the right value). **Selection must have high precision** (choose the one the question asks for). Keep the recall job deterministic and the precision job semantic. ## How Jev makes it cheap [[Jev]]'s Choice primitive is a natural selector: the options are whatever you pass in (up to 255), and the answer is always one of them, with a probability distribution and a confidence. Put the literal candidate spans as options, add `none`, and you have a selector that can't return anything else. Because extra questions are close to free (see [[Speculative Fan-Out]]), you can ask several selections plus attribute questions (currency, country, credit or charge) in the same request. ## Worked examples from the cookbooks **Pre-parsed values.** A regex tuned to over-find proposes candidates; a Choice picks one; code normalizes it. - *Email:* four addresses in a thread, and the body asks for the receipt to go to a personal address instead of the billing alias. The receipt question picked the personal Gmail on the Reply-To line (confidence 0.98); the sender question picked the From address (1.00) - *Phone:* three numbers, none with a country code. Jev picked the mobile (1.00) and read the country from the surrounding text (US, 0.90); the `phonenumbers` library then formatted it as `+14155550177` - *Money:* four amounts on an invoice. The total ($1,315.50) and the credit ($50.00) were picked, and a Noul "is this a credit?" answered 0.01 for the total and 0.99 for the credit, so code knows the sign **Date parts.** Dates are a known weak spot (the Jev 1.13 docs admit it reads dates as text, not as ordered quantities). So the date extraction cookbook never asks for a date. Seven Choice questions in one call read the *parts*: `mode` (absolute / relative / none), `month`, `day`, `year` (1900 to 2050, plus `none` and `out_of_range`), `day_anchor` (today, tomorrow, day after, weekday), `weekday` and `week_offset`. Code assembles the date, infers a missing year, resolves "next Thursday" against a pinned TODAY, and takes the *minimum* confidence over the parts it used. 6/6 correct at 0.91 to 0.97 confidence; a date the form never mentions came back as "absolute date incomplete" at 0.46 and went to review. **Closed-set function calling.** The function calling cookbook reads Python type hints. A `Literal` argument becomes a Choice over exactly the allowed values (the option keys are the strings the function accepts, so nothing maps back). A `list[Literal]` becomes one Noul per member. A `bool` becomes a flag Noul. Free-form arguments (ints, free text) get no question and keep their default. A `__tool__` Choice picks among the 10 functions. "plot rolling correlation between nvda and spy for the past month" became `rolling_correlation(NVDA, SPY, 1mo)` at 0.91. It's [[AI Tool Use]] where invalid arguments are impossible by construction. ## The "was it stated?" gate This is the detail I liked most. A forced choice always names *something*. Ask "which time window?" about "is amd tracking nvidia lately" and you'll get a confident window the user never specified. So every optional argument gets a companion Noul: *does the command say anything about this at all?* If not, the argument is omitted and the function's default applies. For that command, `window` and `resolution` were left out; symbol AMD came back at 0.87 and benchmark NVDA at 0.78. Same logic in the email case: always include `none` as an option. A selector with no way out will crown a winner anyway (see [[Relative vs Absolute AI Judgments]]). ## Gotchas - **Candidate generation is the hard part.** Emails, phone numbers and amounts have good regexes. Names don't. Candidates then have to come from a roster, a named-entity recognizer or an LLM proposer (and an LLM proposer brings the invention risk back, one step earlier) - **255 options max.** With more candidates, narrow in two stages: pick the section first, then the span inside it - **Locale.** `$1,315.50` and `€1.315,50` use opposite separators. Ask a Noul about the number convention and branch in code - **Role wording matters.** Two arguments drawn from the same list (symbol and benchmark) need questions that spell out the roles: "the one being measured, named first" vs "the second one named, the yardstick" - **Write about the idea, not the parameter name.** "Which resolution?" gives the model nothing to match. Describe what the argument means in user terms - **Minimum, not product.** For a call with many arguments, the call's confidence is the weakest judgment. A product shrinks with every argument even when none is shaky ## My take This is the "model reads, code computes" principle made concrete, and I think it's underused. The usual recipe is to ask an LLM for the final JSON and then write validators to catch its mistakes. Candidate-then-select moves the guarantee upstream: the value is correct *by construction* (it's in the source), and you only have to worry about whether the right one was picked, which comes with a confidence you can gate on. It's also a nice counterpoint to [[LLM Structured Outputs]]. Structured outputs guarantee the *shape*. Candidate-then-select guarantees the *provenance*. The SDE cascade shows why that difference matters: a schema-valid fabrication passes every shape check (see [[AI Model Cascades]]). Google's [[LangExtract]] chases the same goal from the other side, by grounding generated extractions back to source spans. ## References - [Pre-parsed value extraction cookbook (TypeSafe docs)](https://docs.typesafe.ai/cookbooks/pre_parsed_value_extraction_cookbook) - [Date extraction cookbook (TypeSafe docs)](https://docs.typesafe.ai/cookbooks/date_extraction_cookbook) - [Function calling cookbook (TypeSafe docs)](https://docs.typesafe.ai/cookbooks/function_calling) - [Jev 1.13 model jaggedness (TypeSafe docs)](https://docs.typesafe.ai/model-jaggedness/jev-1.13) - [Vinyals, Fortunato, Jaitly, Pointer Networks (2015)](https://arxiv.org/abs/1506.03134) - [Rajpurkar et al., SQuAD (2016)](https://arxiv.org/abs/1606.05250) ## Related - [[Atomic Question Decomposition]] - [[Relative vs Absolute AI Judgments]] - [[System One Primitives]] - [[Speculative Fan-Out]] - [[Confidence-Gated Routing]] - [[AI Model Cascades]] - [[LLM Structured Outputs]] - [[LangExtract]] - [[AI Tool Use]] - [[AI Hallucination]] - [[Jev]]