# Speculative Fan-Out Speculative fan-out means asking a model every question your code *might* need about an input in a single request, including questions that only matter on one branch of your logic. When the answers come back, code decides which ones are relevant and ignores the rest. You trade a few cheap extra tokens for zero follow-up round trips. Why does this matter? Because the natural way to write an AI workflow is sequential. First ask "what kind of ticket is this?", then, if it's a bug, ask "how severe?", then maybe "are there repro steps?". Each step waits for the previous one. With an LLM that's seconds per hop, and you pay to send the ticket again every time. Fan-out flips that: ask everything upfront, branch afterwards. ## The general idea This is an old trick from CPUs. Speculative execution computes the result of a branch before knowing whether the branch will be taken, and throws the work away when the guess was wrong. It pays off whenever idle waiting costs more than the wasted work. Databases do something similar with prefetching, and web apps do it when they load the data for the next screen before you click. It only works when two conditions hold: - **The questions are independent.** The answer to one must not depend on the answer to another. If it does, you have a real dependency and you need two calls - **Extra questions are cheap next to the shared context.** If the document dominates the cost and each extra question adds a few tokens, speculating is almost free Plain LLMs fail the second condition in practice. You can ask five questions in one prompt, but the answers then influence each other (the model reads its own earlier answers), and output tokens cost real money. So people split them, and pay for it in latency. ## How Jev makes it cheap [[Jev]] was designed around this pattern. A request is one *state* plus a map of questions. The state is encoded once; every question reads it in parallel; no question sees another question's answer (TypeSafe calls this the independence guarantee). And output tokens are free. The docs put it bluntly: "Asking a question you might not need is close to free." The numbers from the Parallel questions cookbook: 13 questions (8 Nouls, 2 Choices, 3 Scores) about the GDPR Wikipedia article (~54,000 characters), five runs each way. | | 13 separate calls | 1 batched call | |---|---|---| | Cost | $0.006090 | $0.000497 | | Time (serial) | 2.71 s | 0.27 s | That's **12.2x cheaper and 10.0x faster**, with identical answers: 11 of the 13 questions had a run-to-run standard deviation of exactly 0.0 under both strategies. The noise that exists belongs to the question, not to the batching. (Another docs page quotes 11.5x and 9.6x for the same experiment, so two versions of the numbers coexist.) The saving approaches Nx as the document grows, because the document is what you stop paying for N times. ## Worked example: ticket triage The fan-out pattern page uses one support ticket ("charged twice, can't log in, please add Apple Pay, this is getting frustrating") and five questions in one request: - **Choice** `category`: bug_report / billing / feature_request / account - **Score** `bug_severity` (cosmetic / degraded with workaround / blocking), only relevant for bugs - **Noul** `has_reproducible_steps`, only relevant for bugs - **Noul** `refund_requested`, only relevant for billing - **Score** `frustration`, relevant everywhere Then plain code: ```python if category.choice == "bug_report": if bug_severity.score > 1.5 and bug_repro.noul > 0.6: escalate_to_engineering(ticket_id, severity="high") elif category.choice == "billing": if refund.noul > 0.7: route_to_billing_with_flag(ticket_id, refund_likely=True) if frustration.score > 1.5: flag_for_priority_response(ticket_id) ``` If the ticket turns out to be a feature request, the severity answer is simply never read. The function calling cookbook pushes this further: one request per user command carries 54 questions (which of the 10 functions to call, plus the arguments of *every* function), and the dispatcher only reads the chosen function's answers. See [[Candidate-Then-Select Extraction]]. TypeSafe's smart home demo even names the anti-pattern: sequential gating (category, then device, then action) minimizes the number of questions but ends up slower and more expensive than asking everything at once. ## When NOT to fan out Split into two requests only when code *can't build* the second request without the first answer. The docs list three legitimate cases: - **You need to fetch more data.** Skill suggestion ranks 182 skills first, then fetches the full text of the top 3 - **The units don't exist yet.** The autoformat cookbook can only classify blocks after a first pass has stitched lines into blocks - **The next options depend on the previous answer.** Walking a taxonomy (see [[Hierarchical Classification]]) ## Gotchas - **The 10x speed figure sums serial latencies.** Fire 13 calls concurrently and the latency gap narrows a lot. The 13x token cost doesn't - **Speculative questions must still make sense on their own.** A question phrased as if the branch were already known ("What action should be taken on the lights?") gets answered anyway on an input that has nothing to do with lights. That's fine as long as code never reads it on that branch - **Watch the context limit.** 64k tokens per request for state plus all questions. Hundreds of speculative questions with long criteria can eat into that - **Independence cuts both ways.** Because questions can't see each other, you can't ask "given that it's a bug, how severe?". Write each question so it stands alone ## My take I like this pattern because it inverts a habit I have with LLMs. With chat models I instinctively minimize the number of questions, because each one costs seconds and money. With a decision model, the right instinct is the opposite: ask MORE, then let code be picky. It's the same shift as going from hand-rolled SQL queries per screen to fetching a denormalized record once. The discipline it demands is independence. If you catch yourself writing a question that only makes sense given another answer, that's either a real dependency (two calls) or a sign the question should be rewritten to stand alone. That's also where fan-out meets [[Atomic Question Decomposition]]: small, independent questions are exactly what makes fanning out possible. ## References - [Speculative fan-out (TypeSafe docs)](https://docs.typesafe.ai/patterns/fan-out) - [Parallel questions cookbook (TypeSafe docs)](https://docs.typesafe.ai/cookbooks/parallel_questions) - [Function calling cookbook (TypeSafe docs)](https://docs.typesafe.ai/cookbooks/function_calling) - [How to build with System One (TypeSafe docs)](https://docs.typesafe.ai/concepts/how-to-build-with-system-one) - [Smart home demo (TypeSafe docs)](https://docs.typesafe.ai/demos/smart-home) - [Speculative execution (Wikipedia)](https://en.wikipedia.org/wiki/Speculative_execution) ## Related - [[Jev]] - [[System One Models]] - [[System One Primitives]] - [[Atomic Question Decomposition]] - [[Composite Scoring]] - [[Confidence-Gated Routing]] - [[Candidate-Then-Select Extraction]] - [[Hierarchical Classification]] - [[AI KV Cache]] - [[AI Cost Management]]