# Composite Scoring Composite scoring is a way to rank things on several criteria at once. Instead of asking one big question ("how good is this candidate?"), you score each dimension separately, normalize every score to the same 0 to 1 range, and combine them with weights you write in code. The model judges; code decides how much each judgment counts. The reason I care: when you ask an LLM for one overall rating, all the trade-offs happen inside the model, where you can't see or tune them. If the ranking looks wrong, your only lever is rewriting the prompt and hoping. With a composite score, every factor stays visible, and fixing the ranking means changing a number. ## The general idea Nothing new here, and that's a good thing. This is the **weighted sum model** from multi-criteria decision analysis, formalized by Peter Fishburn in 1967: multiply each criterion's value by its weight and add them up. It's the logic behind every scoring rubric, every lead-scoring spreadsheet, every "decision matrix" you've drawn on a whiteboard. Credit scores, hiring scorecards and search ranking functions all work the same way. It has one famous constraint: the criteria must be on the same scale. Otherwise, as the Wikipedia article puts it, you're adding apples and oranges. That's why normalization is part of the pattern, not an optional step. What always made it expensive with AI is that each criterion needs its own judgment, and each judgment used to mean another LLM call. Four criteria times 10,000 resumes is 40,000 calls. So people collapsed everything into one "overall score" prompt and lost the transparency. ## How Jev makes it cheap With [[Jev]], every criterion is a Score question, and all of them go in the same request (see [[Speculative Fan-Out]]). Adding a dimension costs a few tokens, not a new call. Each Score comes back as the expected value over the levels you described, plus the full distribution and a confidence (see [[System One Primitives]]). And because the combination happens in code, the same request can feed several different rankings. ## Worked example: resume screening The composite scoring page asks four 5-level Score questions about each resume: Python depth, team leadership, system design, and generalist ability. Each level describes a concrete situation ("Primary language, multiple projects", "Managed a team with direct reports"), not a degree. Then: ```python py = answers["python_depth"].score / 4 # 5 levels: 0..4 lead = answers["team_leadership"].score / 4 arch = answers["system_design"].score / 4 general = answers["generalist"].score / 4 ic_score = 0.40*py + 0.10*lead + 0.40*arch + 0.10*general # senior IC em_score = 0.15*py + 0.40*lead + 0.20*arch + 0.25*general # eng manager ``` One request per resume, two role rankings. If the top candidates for the manager role don't match what the hiring team expects, you change 0.40 to 0.30. You don't touch a prompt. A smaller example from the docs, ticket priority: `0.6 * severity + 0.3 * frustration + 0.1 * report_quality` = 0.6 × 0.62 + 0.3 × 0.64 + 0.1 × 1.0 = **0.664**. ## Gotchas - **Normalize first.** A 3-level Score (0 to 2) and a 4-level one (0 to 3) aren't comparable raw. Divide each by its top level (`len(levels) - 1`) so the weights mean what they say - **One dimension per Score.** "Punctual and smart and experienced" is three questions. Mixed dimensions produce low confidence and meaningless middle values - **Read the distribution, not only the score.** A Score of 1.0 can mean "all on level 1" or "half on 0, half on 2". Those are very different candidates. Low confidence is a signal the levels overlap or the input doesn't say enough - **Levels describe situations, not degrees.** Jev judges each level separately and never sees its number, so `["0", "1", "2"]` gave a split 0.45/0.55 answer at confidence 0.33 where descriptive levels gave 0.0 at confidence 1.0 - **Weights are opinions.** A weighted sum looks objective because it's arithmetic. It isn't. The weights encode what you value, so write them down, review them, and keep them in one place - **Linear isn't always right.** A weighted sum lets a great score on one criterion compensate for a terrible one on another. If a criterion is a hard requirement (must know Python), make it a filter or a Noul gate before the sum, not a weight - **Calibration doesn't compose for free.** Each Score may be calibrated; the weighted combination isn't a calibrated probability of anything. Treat it as a ranking key ## Going further: learn the weights When you have labeled outcomes, you can stop guessing weights. Feed the scores (and their confidences) as features into a regression or gradient-boosted model and let it find the weights. That's where composite scoring turns into [[LLM-Generated Features for Classical ML]]. Rajesh Beri's phishing test is a good illustration: one "is this phishing?" question gave 62.6% accuracy, five atomic questions combined by a logistic regression gave 95.0%. His own verdict is worth keeping in mind though: "The 95% is not Jev. It is Jev plus your labelled data plus a regression you maintain." ## My take This is [[Atomic Question Decomposition]] applied to ranking, and I think it's the right default for any "score these items" task. What I like most is the change in who owns the judgment. The model answers narrow, checkable questions ("how much leadership experience is in this resume?"), and the business logic ("for this role, leadership matters 40%") lives in code where a human can read, argue about and version it. My only warning: don't let the arithmetic fool you into thinking the result is neutral. Weights are decisions. Make them explicit, and you at least get to have the argument in the open. ## References - [Composite scoring (TypeSafe docs)](https://docs.typesafe.ai/patterns/composite-scoring) - [Score primitive (TypeSafe docs)](https://docs.typesafe.ai/primitives/score) - [How to build with System One (TypeSafe docs)](https://docs.typesafe.ai/concepts/how-to-build-with-system-one) - [Weighted sum model (Wikipedia)](https://en.wikipedia.org/wiki/Weighted_sum_model) - [Rajesh Beri on Jev calibration and decomposition](https://www.beri.net/article/typesafe-jev-typed-decision-model-calibration-decomposition-shadow-eval) ## Related - [[Atomic Question Decomposition]] - [[System One Primitives]] - [[Speculative Fan-Out]] - [[LLM-Generated Features for Classical ML]] - [[Confidence-Gated Routing]] - [[Reranking]] - [[Expected Utility Theory]] - [[Jev]] - [[Decision Models (DMs)]]