# AI Post-Training Post-training is everything you do to a language model after pretraining. Pretraining gives you a base model: a next-token predictor that has read a big chunk of the internet and can continue any text. It knows a LOT, but it doesn't do anything useful on request. Ask a base model a question and it may answer, or it may write three more questions in the same style, because that's what the next tokens of a quiz page look like. Post-training is where the model gets a job. And I think it's the most underrated part of the whole story, because it decides what the model is optimized FOR. Same pretrained weights, different post-training, and you get a chatbot, a maths olympiad solver, or (in [[TypeSafe AI]]'s case) a [[Decision Models (DMs)|decision model]] like [[Jev]]. ## The usual stack Most frontier labs run some variation of this sequence: 1. **Supervised fine-tuning (SFT).** Train on curated examples of the behavior you want (question and good answer pairs, conversations). See [[AI Fine-Tuning]] and [[AI Instruction Tuning]]. For [[InstructGPT]], that was about 13,000 prompts with answers written by labelers 2. **Preference optimization.** Show humans (or another model) several outputs, ask which one is better, and push the model toward the winners. That's [[Reinforcement Learning From Human Feedback (RLHF)|RLHF]] with PPO, or one of its cheaper descendants like DPO (which skips the separate reward model), or RLAIF and [[Constitutional AI]] (where AI feedback replaces some of the human feedback) 3. **Reinforcement learning on checkable tasks.** Since late 2024, labs add [[Reinforcement Learning with Verifiable Rewards (RLVR)|RLVR]]: generate solutions to maths or coding problems, check them with a program, reward the correct ones. This is what produced [[AI Reasoning Models|reasoning models]] like o1 and DeepSeek-R1 4. **Safety and polish passes.** Refusal training, style, tool use, persona, format rules Tülu 3 (Lambert et al., AI2, November 2024, arXiv:2411.15124) is a good public reference if you want to see a full open recipe: SFT, then DPO, then RLVR, with the data and code published. ## Name the target, not the algorithm Here's a framing I picked up from Diogo Almeida on the Latent Space podcast (September 2026) that made things click for me. When he says "RLHF", he means the *task* (instruction following, optimizing for what human raters prefer), not the PPO algorithm: "It's not about the PPO. That part doesn't matter." DPO and its descendants "also do RLHF, despite not using the algorithm." That gives you a cleaner map of post-training, organized by what gets rewarded: | Approach | What gets rewarded | What you get | |---|---|---| | [[Reinforcement Learning From Human Feedback (RLHF)\|RLHF]] | Outputs humans prefer | Chat assistants ([[ChatGPT]], [[Claude]]) | | [[Reinforcement Learning with Verifiable Rewards (RLVR)\|RLVR]] | Answers a program can verify as correct | [[AI Reasoning Models\|Reasoning models]] (o1, DeepSeek-R1) | | [[Reinforcement Learning for Calibrated Decisions (RLCD)\|RLCD]] | Probabilities that match reality | [[System One Models]] ([[Jev]]) | The algorithm (PPO, GRPO, DPO, plain supervised learning) is a detail. The target decides the behavior. "You get what you optimize for," as Almeida puts it. ## Post-training has side effects Every post-training objective is a proxy, and optimizing a proxy hard enough bends the model in ways nobody asked for ([[Goodhart's Law]]). The well-documented side effects: - **Calibration damage.** The [[GPT4|GPT-4]] technical report (arXiv:2303.08774) measured the pretrained model at an expected calibration error of 0.007 on an MMLU subset, and 0.074 after PPO post-training. OpenAI's own words: "post-training hurts calibration significantly." See [[AI Model Calibration]] - **Diversity loss.** RLHF narrows the output distribution (Kirk et al., arXiv:2310.06452). TypeSafe calls it mode dropping; most people say [[Mode Collapse]] - **[[AI Sycophancy]].** Raters like being agreed with, so the model learns to agree (Sharma et al., Anthropic, arXiv:2310.13548) - **[[Reward Hacking]].** The policy finds what the reward model likes rather than what the humans meant. Gao, Schulman and Hilton (arXiv:2210.10760) showed the true reward first rises then falls as you optimize the proxy harder - **An "alignment tax".** The InstructGPT paper already measured regressions on public NLP benchmarks after RLHF, and mixed pretraining data back in to limit them - **Jaggedness.** RLVR makes models superhuman on the checkable stuff and weird on everything else ([[Jagged Intelligence]]) The pattern I see: pretraining is a strictly proper objective (next-token cross-entropy rewards honest probabilities, see [[Proper Scoring Rules]]). Post-training then swaps that honest objective for a proxy (human approval, pass/fail), and the model's probabilities stop meaning what they used to mean. ## Why it matters for Jev TypeSafe's whole bet is a post-training bet. Almeida's premise (as his AI Engineer speaker bio puts it) is that pretrained models already contain a lot of capability, and the job is extracting it with the right objective and interface. Instead of post-training toward human preference, they post-train toward calibrated decisions over typed answers, on 100% synthetic data (per Almeida on Latent Space). They call themselves "a data lab, not a model lab" for that reason: in post-training, the data IS the objective. ## My take I think "post-training" is where the interesting AI work moved around 2023. Pretraining is a race of compute budgets few players can join. Post-training is where you decide what the model is for, and small labs can still compete there. The flip side: every chatbot you use carries the fingerprints of the preference data it was tuned on, and it's worth knowing that when you ask it how sure it is. ## References - [Training language models to follow instructions with human feedback (InstructGPT, arXiv:2203.02155)](https://arxiv.org/abs/2203.02155) - [Tülu 3: Pushing Frontiers in Open Language Model Post-Training (arXiv:2411.15124)](https://arxiv.org/abs/2411.15124) - [GPT-4 Technical Report (arXiv:2303.08774)](https://arxiv.org/abs/2303.08774) - [Understanding the Effects of RLHF on LLM Generalisation and Diversity (arXiv:2310.06452)](https://arxiv.org/abs/2310.06452) - [Scaling Laws for Reward Model Overoptimization (arXiv:2210.10760)](https://arxiv.org/abs/2210.10760) - [TypeSafe AI primer](https://docs.typesafe.ai/introduction/machine-learning-primer) - [Jev: System One models for Prod, not God, with Diogo Almeida (Latent Space)](https://www.latent.space/p/jev) ## Related - [[Reinforcement Learning From Human Feedback (RLHF)]] - [[Reinforcement Learning with Verifiable Rewards (RLVR)]] - [[Reinforcement Learning for Calibrated Decisions (RLCD)]] - [[InstructGPT]] - [[AI Fine-Tuning]] - [[AI Instruction Tuning]] - [[AI Model Calibration]] - [[Mode Collapse]] - [[Reinforcement Learning (RL)]] - [[Large Language Models (LLMs)]]