# InstructGPT InstructGPT is the [[OpenAI]] model (and paper) that turned GPT-3 from a text continuer into something that follows instructions. It's the direct ancestor of [[ChatGPT]], and the paper where [[Reinforcement Learning From Human Feedback (RLHF)|RLHF]] went from a research idea to the standard recipe for building assistants. The paper is *Training language models to follow instructions with human feedback* (Ouyang et al., arXiv:2203.02155, submitted 4 March 2022, NeurIPS 2022). [[Diogo Almeida]], who later founded [[TypeSafe AI]], is the fourth author and one of the paper's primary authors (marked as such in the author list). That link is why this note exists in my vault: Jev is, in a way, its author's answer to what InstructGPT started. ## The problem it solved GPT-3 (2020) was impressive but awkward. Ask it "Explain the moon landing to a 6 year old" and it might continue with more prompts like that one, because on the internet, a list of questions is followed by more questions. As the abstract puts it: "Making language models bigger does not inherently make them better at following a user's intent." The model was capable, but not aligned with what users wanted. ## The recipe (three steps) 1. **Supervised fine-tuning (SFT).** About 40 contractors (hired via Upwork and Scale AI) wrote demonstrations of good answers. About 13,000 training prompts, from the OpenAI API and written by labelers. GPT-3 was fine-tuned on them 2. **Reward model.** Labelers ranked between K = 4 and 9 model outputs per prompt, from best to worst. A 6B-parameter reward model learned to predict those rankings (33,000 training prompts) 3. **Reinforcement learning.** The SFT model was optimized with PPO (Schulman et al., arXiv:1707.06347) against the reward model on 31,000 API prompts. A per-token KL penalty kept the policy close to the SFT model, which limits drift and [[Reward Hacking]]. A variant called PPO-ptx mixed pretraining gradients back in to reduce regressions on public NLP benchmarks (what the paper calls the "alignment tax") Almost every chat assistant since has used some version of this pipeline (see [[AI Post-Training]]). ## The headline results - **Small beat big.** Outputs from the 1.3B-parameter InstructGPT were preferred to those of the 175B GPT-3, "despite having 100x fewer parameters" - The 175B InstructGPT was preferred to 175B GPT-3 85 ± 3% of the time, and 71 ± 4% of the time against few-shot prompted GPT-3 - It made things up less often on closed-domain tasks (a 21% vs 41% hallucination rate) - Some improvement in truthfulness and less toxic output, with "minimal performance regressions on public NLP datasets" The "small beats big" result is the one that mattered most, in my opinion. It showed that post-training could be worth more than a 100x bigger model. That's been true ever since. ## Where RLHF came from (it wasn't invented here) InstructGPT didn't invent RLHF. The idea goes back to Christiano, Leike, Brown, Martic, Legg and [[Dario Amodei]] (*Deep reinforcement learning from human preferences*, arXiv:1706.03741, 2017), who taught agents Atari and simulated robot tasks from human comparisons of short clips, with feedback on less than 1% of interactions. Ziegler et al. (2019, arXiv:1909.08593) and Stiennon et al. (*Learning to summarize from human feedback*, 2020, arXiv:2009.01325) applied it to language. InstructGPT scaled it to general instruction following, and ChatGPT (November 2022) put it in front of the world. So when TypeSafe says Almeida "co-invented RLHF", read it as marketing shorthand. Co-author of the paper that made RLHF THE recipe for assistants, yes. Inventor of the idea, no. ## What the paper already warned about Read today, the paper already names many of the problems that came later: - The model optimizes for what ~40 labelers preferred, following OpenAI's instructions. The authors say plainly it's aligned to "the stated preferences of a specific group of people (mostly our labelers and researchers)", not to any broader notion of human values - The alignment tax was reduced, not eliminated - InstructGPT "still makes simple mistakes": it follows false premises, hedges too much on simple questions, and can follow harmful instructions What it didn't measure was calibration. The [[GPT4|GPT-4]] report a year later did, and found that post-training pushed the expected calibration error from 0.007 to 0.074 (see [[AI Model Calibration]]). Preference optimization also narrows output diversity ([[Mode Collapse]]) and rewards agreeable answers ([[AI Sycophancy]]). ## The Jev connection Almeida's view today is that this recipe, perfect for chat, sent the industry down a "weird detour". His words: "Why do all LLMs require a human in the loop? The simple answer is we literally put them in the loop." We trained models to please the humans reading their output, and then wondered why they need humans supervising them. [[Jev]] is the opposite bet: post-train for [[Reinforcement Learning for Calibrated Decisions (RLCD)|calibrated decisions]] that software can consume, and optimize for "prod, not God". Whether or not you buy the pitch, I find it a great story: one of the people who built the chatbot recipe spending four years arguing it was the wrong recipe for automation. ## My take InstructGPT is one of the most influential ML papers of the decade, and it's very readable. If you want to understand why assistants behave the way they do (the eagerness to help, the confident tone, the hedging), this paper is the place to start. The second lesson I take from it: the objective you choose in post-training shapes everything downstream. Pick "what humans prefer" and you get something humans like talking to. That's not the same as something you can trust unattended. ## References - [Training language models to follow instructions with human feedback (arXiv:2203.02155)](https://arxiv.org/abs/2203.02155) - [Deep reinforcement learning from human preferences (arXiv:1706.03741)](https://arxiv.org/abs/1706.03741) - [Fine-Tuning Language Models from Human Preferences (arXiv:1909.08593)](https://arxiv.org/abs/1909.08593) - [Learning to summarize from human feedback (arXiv:2009.01325)](https://arxiv.org/abs/2009.01325) - [Proximal Policy Optimization Algorithms (arXiv:1707.06347)](https://arxiv.org/abs/1707.06347) - [GPT-4 Technical Report (arXiv:2303.08774)](https://arxiv.org/abs/2303.08774) - [TypeSafe AI team](https://typesafe.ai/team) - [Diogo Almeida on ChatGPT, RLHF, and TypeSafe AI (explainx.ai)](https://www.explainx.ai/blog/diogo-almeida-typesafe-ai-rlhf-detour-profile-2026) ## Related - [[Reinforcement Learning From Human Feedback (RLHF)]] - [[Diogo Almeida]] - [[ChatGPT]] - [[OpenAI]] - [[AI Post-Training]] - [[AI Instruction Tuning]] - [[AI Model Calibration]] - [[Mode Collapse]] - [[AI Sycophancy]] - [[TypeSafe AI]] - [[Jev]]