Original InstructGPT deep reading

Series · 1 posts

← Back to Paper Reading

  • InstructGPT: Align Instructions with Human Feedback, but 2022 Preference Win Rates Do Not Represent Later ChatGPT

    Intermediate Build the foundations first
    Original InstructGPT deep reading: Part 1 , Note: Aug 28, 2026 , Paper: 2022 , NLP

    A source-grounded reading of Ouyang et al., NeurIPS 2022 / arXiv:2203.02155: on a frozen GPT-3 architecture, a three-stage pipeline—SFT demonstrations, a reward model, and PPO (default PPO-ptx)—aligns a pretrained LM to human preferences. 175B InstructGPT is preferred over 175B GPT-3 85±3% of the time; this is 2022 API labeler evidence, not a ChatGPT product SLA, GPT-4 eval, or DPO contract.

    Understand it in 90 seconds
    Problem
    Bigger LMs do not automatically follow user intent better. GPT-3-class models under a next-token pretraining objective often produce untruthful, toxic, or unhelpful outputs (Section 1). Language modeling is misaligned with responding helpfully and safely to instructions.
    Core insight
    On a frozen GPT-3 architecture, run three stages: (1) SFT—fine-tune on demonstrations from about 40 contractors; (2) RM—train a 6B reward model on human rankings; (3) PPO—optimize the policy against RM scores with a per-token KL penalty toward SFT; the default PPO-ptx mixes pretraining gradients to reduce public NLP regressions (Figure 2, Equation 2, Section 3.5). The control point is human-preference alignment, not a new attention stack.
    Strongest evidence
    Labeler preference evals on the API prompt distribution (Figure 1, Section 4.1): 175B InstructGPT is preferred to 175B GPT-3 85±3% of the time; to few-shot GPT-3 71±4%; 1.3B InstructGPT still beats 175B GPT-3 (>100× parameter gap). Versus the 175B SFT baseline, InstructGPT wins 73.4±2%, beating FLAN/T0 fine-tunes (26.8±2%, 29.8±2%).
    Main boundary
    Closed 2022 GPT-3 family models; preferences come from a specific labeler pool and API Playground distribution (Sections 5.2–5.3). ChatGPT product metrics, GPT-4, DPO, Llama-2-chat, and Constitutional AI are outside this PDF; YOLO mAP and Transformer WMT BLEU are not alignment contracts.
    Read the full deep dive

For speaking invitations, internal engineering sessions, or architecture exchange, see the topics and public work I can bring into the conversation.

Speaking & contact