# LLM-Generated Features for Classical ML Classical [[Machine Learning (ML)|machine learning]] models like gradient boosting or [[Logistic Regression|logistic regression]] are fast, cheap, explainable and really good at learning from labeled tables. They're useless on raw text, though, because a tasting note or a support ticket isn't a table of numbers. The idea in this note is to build that table with an AI model: ask it many typed questions about each text ("how positive is the overall tone?", "does it mention a single vineyard?"), use each answer as a numeric column, and train the classical model on those columns against the labels you care about. You get the language understanding of a big model and the rigor of supervised learning: a held-out test set, feature importances, a model you can retrain in seconds, and no prompt to babysit. [[TypeSafe AI]] lists this as step 7 of how to build with [[Jev]] ("combine outputs in code, or feed them to a classical ML model"), and published a full cookbook for it. The same pattern works with any model that returns probabilities or scores. ## How the questions become columns - **Presence questions** ("does the note mention oak?") use a yes/no probability, i.e. a Noul in Jev's terms → **1 column** - **Intensity questions** ("how intense is the fruit?") use a 5-level Score → **2 columns**: the expected level *and* the spread of the distribution. The spread is a free "how unsure was the model" feature, and it carries signal of its own See [[System One Primitives]] for how Nouls and Scores work. ## The TypeSafe cookbook: wine reviews Task: predict a critic's score (80 to 100) from the tasting note. 2,000 reviews: 1,200 for development, 800 held out and scored once at the end. Claude Sonnet 5 proposed the questions, Jev (`jev-1.12`) answered them, CatBoost trained on the answers. | How the note becomes a score | Held-out RMSE | Spearman | |---|---|---| | Predict the average score | 3.088 | -0.01 | | CatBoost on word counts | 2.466 | 0.61 | | Ask the model for the score directly (one 10-level Score, level 0 = 80 ... level 9 = 100, plus one learned offset) | 2.145 | 0.76 | | 18 questions from one proposal, no loop | 1.869 | 0.78 | | **38 questions after 5 rounds of the loop** | **1.772** | **0.80** | (RMSE is the typical error in critic-score points; lower is better.) The row I find most telling is the third one. Asking the model for the target *directly* is worse than asking it many small questions and letting a classical model learn how they combine. It's [[Atomic Question Decomposition]] with the weights learned from data instead of written by hand. The final 38 questions (29 Scores, 9 Nouls) produced 67 columns. The top feature was the overall tone positivity at 17.4% of importance; a Noul about single-vineyard or prestige signals came 4th at 7.2%. ## The autoresearch loop TypeSafe calls the improvement process an "autoresearch loop". Nobody writes the questions by hand: 1. **Round 1:** the LLM reads 60 notes spread across the score range and proposes questions 2. Jev answers every question for every row (one request per row, all questions in it) 3. CatBoost trains with k-fold cross-validation on the dev rows 4. **Later rounds:** the LLM reads the 30 worst-predicted and 30 best-predicted notes (with current and previous predictions) plus the feature importances, and returns up to 18 actions: add, revise or drop a question 5. **Acceptance rules:** new questions stay unless their column is flat; revisions and drops are kept only if the cross-validated error improves (refits cost no API calls, so rejecting a change is free) What happened: rounds 2 to 5 improved the held-out error by only **0.097 points** (95% CI -0.147 to -0.050). **Most of the gain came from the very first proposal.** Round 5 was mostly drops (4 adds, 2 rewordings, 8 drops) and was the first round with no dev improvement, i.e. a plateau. ## Lessons worth keeping - **Don't pre-filter questions.** A question that applies to one row in ten looks useless in a 60-note sample but can be the most useful column. Answer it everywhere and let the model decide - **Cost scales with rows, not questions.** 100,000 rows means 100,000 requests per round, and a revised question means another pass over every row. Extra questions in the same request are nearly free - **Rate limits come first.** Eight parallel workers already hit the limit on a shared key - **Expect diminishing returns from iteration.** Budget for one good proposal plus a couple of refinement rounds, and stop on a plateau ## The same pattern, elsewhere Rajesh Beri's phishing test is a small version of the same thing: five atomic questions per email combined with a logistic regression took Jev from 62.6% (one question) to 95.0% accuracy. His warning applies to this whole note: "The 95% is not Jev. It is Jev plus your labelled data plus a regression you maintain." That's the deal. You need labels, and you own a model. And if you don't have labels, TypeSafe's suggestion is to generate them with an ensemble of expensive reasoning models, then train the cheap pipeline on them. ## Ideas to extend it From the cookbook's next steps: screen proposed questions *with Nouls about the question itself* (answerable from the text? unambiguous? applies to most rows? varies across rows?), prune correlated features, mix proposer model families, compare against TF-IDF and embedding baselines ([[Embeddings]]), use chronological or grouped splits when that matches deployment, and check stability across seeds. ## Limits - **One dataset, one run.** It's a convincing demo, not a benchmark - **Your features depend on a model version.** If the provider updates the model, your columns shift and your trained model drifts. Pin the version (e.g., `jev-1.13.0`) and re-validate on upgrades - **Leakage is easy.** The proposer reads dev examples and their scores; keep the held-out set out of the loop entirely (the cookbook does) - **Importance is split.** A Score question owns two columns, so you have to sum their importances to judge the question ## My take This is one of the most practical ideas I took from the TypeSafe docs, and it isn't really about Jev. It reframes "use AI to predict X" into "use AI to *describe* the input, then use statistics to predict X". That split plays to each side's strengths: the language model reads, the classical model learns what actually correlates with the outcome. And the LLM-in-the-loop part turns feature engineering (a tedious, expert job) into something closer to a search you can run overnight. ## References - [Autoresearch feature discovery cookbook (TypeSafe docs)](https://docs.typesafe.ai/cookbooks/autoresearch_feature_discovery) - [How to build with System One (TypeSafe docs)](https://docs.typesafe.ai/concepts/how-to-build-with-system-one) - [TypeSafe Jev: typed decision model, calibration, decomposition (Rajesh Beri)](https://www.beri.net/article/typesafe-jev-typed-decision-model-calibration-decomposition-shadow-eval) - [CatBoost](https://catboost.ai/) ## Related - [[Atomic Question Decomposition]] - [[System One Primitives]] - [[Composite Scoring]] - [[Jev]] - [[Machine Learning (ML)]] - [[Logistic Regression]] - [[Embeddings]] - [[AI Evaluation]]