Introduction: Bigger Is Not Better
When GPT-3 debuted with 175B parameters in 2020, the world was stunned. But users quickly discovered an awkward truth: this behemoth often did more harm than good. Ask it for a code comment and it produces a blog post. Ask a factual question and it confidently hallucinates. Tell it to refuse inappropriate requests and it spills harmful content after a few leading prompts.
The root cause is misalignment — the gap between the model’s language-modeling objective (“predict the next token”) and what users actually want.
In March 2022, OpenAI published a paper — “Training Language Models to Follow Instructions with Human Feedback” — that would reshape the entire industry. The method is called RLHF (Reinforcement Learning from Human Feedback), and it transforms raw GPT-3 into InstructGPT through three stages. ChatGPT’s birth, later that year, was the direct continuation of this line.
This article dissects each stage, key hyperparameters, quantitative results, and the “alignment tax” problem.
Step 1: Supervised Fine‑Tuning — Teaching the Model What a Good Answer Looks Like
The first step is straightforward: hire human labelers to write high-quality answers, then fine-tune the model to mimic them.
OpenAI hired contractors to write demonstrations for prompts collected from the API — across generation, QA, writing, summarization, coding, and more. The bar is not “perfect,” but “helpful”: answer the question, don’t fabricate, don’t give harmful advice.
The result: roughly 13,000 labeled demonstrations (12,725 training + 1,653 validation), plus a small slice (~1.4k) from real API users as supplementary data.
GPT‑3 is fine-tuned on this data with supervised learning. 1.3B and 6B models train for 16 epochs with batch size 32, learning rate 9.65e-6; the 175B variant uses batch size 8 and learning rate 5.03e-6. All models use residual dropout 0.2 and cosine LR decay. The checkpoint with the highest RM score on validation is selected.
Figure 1: The three-step RLHF pipeline — SFT learns from human demonstrations, Reward Model learns from human preference rankings, PPO optimizes the policy with the learned reward signal
This produces a model π_SFT that “can answer” but isn’t yet finely aligned.
Step 2: Reward Model — Teaching the Model to Distinguish Good from Bad
SFT only teaches the model how to write, not which style of writing humans prefer. Step 2 trains a scorer to predict which of two responses a human will prefer.
For a batch of prompts, the SFT model samples K = 4–9 different responses (varying temperature), and labelers rank them from best to worst. Each prompt’s ranking yields C(K, 2) comparison pairs, turning the labeling problem from absolute scoring into relative preference.
This is a brilliant design decision: pairwise comparison produces higher inter-annotator agreement than absolute 1–5 scoring. Two labelers may disagree on whether a response is a “3” or a “4,” but they almost always agree on which of two responses is better.
The reward model is trained on these comparisons. About 33,000 prompts for training. A crucial finding: the reward model is only 6B parameters — trying 175B led to severe training instability. Single epoch, learning rate 9e-6, batch size 64 (each batch can hold up to 2,304 comparison pairs).
The loss function is intuitive: given a better response y_w and a worse response y_l, maximize the scoring gap:
$$L(θ) = -\log(σ(r_θ(x, y_w) - r_θ(x, y_l)))$$Where σ is sigmoid. Larger gap means higher confidence. The final reward values get a normalization bias so that labeled demonstrations score 0 on average — this establishes a reference point for the PPO stage.
Step 3: PPO Optimization — Driving Alignment with a Reward Signal
With a reward model in hand, Step 3 uses it to optimize the language model policy.
PPO (Proximal Policy Optimization) is a stable policy gradient algorithm. OpenAI’s RLHF implementation uses the original PPO from Schulman et al. (2017) with minor modifications.
Each episode samples a prompt, the current policy generates a response y, and then:
- Positive signal: reward model score r_θ(x, y) — higher is better
- Negative constraint: KL divergence penalty if the policy strays too far from the SFT model
The KL penalty is critical. Without it, the model quickly learns to game the reward model — producing grammatically garbled output that somehow scores high (reward hacking). KL anchors the policy near the SFT distribution, preventing this degradation.
The objective:
$$\max_φ \mathbb{E}[r_θ(x, y)] - β \cdot \text{KL}(\pi_φ \| \pi_{\text{SFT}})$$KL penalty coefficient β = 0.02. OpenAI also experimented with PPO-ptx, mixing a pretraining gradient term (coefficient γ = 27.8) with 10% pretraining data. PPO-ptx substantially reduces the “alignment tax.”
Key PPO hyperparameters: batch size 512, minibatch 64 (single inner epoch only), PPO clip ratio 0.2, constant learning rate, sampling temperature 1 (no decay), EMA weight decay 0.992. Value function is initialized from the reward model at 6B parameters. A total of 256k episodes (~31k unique prompts).
Striking Results
The most dramatic result is from human evaluations:
| Comparison | Win Rate |
|---|---|
| 175B InstructGPT vs 175B GPT-3 | 85% ± 3% preferred InstructGPT |
| 175B InstructGPT vs few-shot 175B GPT-3 | 71% ± 4% |
| 1.3B InstructGPT vs 175B GPT-3 | InstructGPT wins (<1% of the parameters) |
That last row encapsulates the paper’s central message: users would rather use a 1.3B aligned model than a 175B misaligned one. Alignment trumps scale.
On specific dimensions:
- Instruction following: InstructGPT is better at completing user requests (“doesn’t ignore constraints,” “no padding”)
- Truthfulness: On TruthfulQA, InstructGPT produces truthful and informative answers roughly 2× more often than GPT-3; hallucination rate on closed-domain questions drops from 41% to 21%
- Toxicity: When prompted to be respectful, InstructGPT produces about 25% less toxic output than GPT-3
- Bias: No meaningful improvement on Winogender or CrowS-Pairs benchmarks
The toxicity improvement — while real — is explicitly called out by the authors as modest. RLHF is not a silver bullet for safety.
The Alignment Tax: Trade-offs in Capability
RLHF introduces what the authors call the alignment tax: performance regression on certain NLP benchmarks (e.g., DROP QA and translation tasks). The PPO-ptx variant significantly mitigates this.
More importantly, InstructGPT generalizes to held-out labelers and to non-English instructions. Labelers who didn’t produce any training data still prefer its outputs, and it handles non-English and code tasks well. This suggests the model is learning a broadly useful notion of “helpfulness,” not just overfitting to specific annotators.
Figure 2: Key metrics — 85% human preference win rate, hallucination rate dropped from 41% to 21%, toxicity reduced by 25%
Guardrail Perspective
As an AI safety practitioner, I see two important takeaways from InstructGPT’s approach:
The positives:
- Ranking beats scoring. Pairwise comparison drastically improves inter-annotator agreement, reducing noise in training data
- KL penalty is a built-in guardrail. It prevents the model from abandoning language ability just to game the reward model — an architectural defense against reward hacking
- PPO-ptx shows alignment and capability can coexist. Mixing pretraining gradients nearly eliminates the alignment tax, proving that safety optimization need not degrade performance
The challenges:
- Limited toxicity reduction. The ~25% decrease is real but leaves the model vulnerable to escaped prompts
- Reward model blind spots. Trained only on “which response is better” data, the RM cannot assess subtle discrimination or indirect harm
- Labeler bias baked into alignment. If the labeler pool carries cultural biases, the ranking data does not represent universal values. The authors acknowledge this as a primary limitation
My assessment: InstructGPT defines a technical roadmap but leaves one question open: who decides what “good” means? RLHF outsources this to labelers, and labeler preferences are not a proxy for universal value. This explains why later work — Anthropic’s Constitutional AI, DeepSeek-R1’s rule-based rewards — repeatedly tries to answer the same question: alignment targets should not be pure statistical preference aggregation.
Conclusion
InstructGPT’s contribution isn’t algorithmic novelty (PPO and RLHF both predate it). It’s the first demonstration that alignment matters more than scale.
| Dimension | Key Finding |
|---|---|
| Methodology | SFT → Reward Model → PPO three-step pipeline, with carefully designed data scales and hyperparameters |
| Key trade-off | KL penalty β = 0.02 balances alignment and capability; PPO-ptx reduces alignment tax |
| Performance | 1.3B aligned model outperforms 175B raw model; hallucination rate halved |
| Limitations | Modest toxicity improvement, no bias improvement, alignment bounded by labeler preferences |
In retrospect, this paper is the technical foundation of ChatGPT and the entire alignment / post‑training era. Subsequent RLHF variants — InstructGPT → ChatGPT → Constitutional AI → DPO → R1’s GRPO — all extend the framework it established. But its core insight remains unchanged: making a model small is easy. Making a model good is what’s worth doing.
References
- Original paper: Ouyang, L., Wu, J., Jiang, X. et al. Training Language Models to Follow Instructions with Human Feedback. NeurIPS 2022. https://arxiv.org/abs/2203.02155
- OpenAI blog: Aligning Language Models to Follow Instructions. https://openai.com/index/instruction-following/
- Technical review: ArthurChiao. InstructGPT Paper Reading Notes. http://arthurchiao.art/blog/instructgpt-paper-zh/
- PPO algorithm: Schulman, J. et al. Proximal Policy Optimization Algorithms. 2017. https://arxiv.org/abs/1707.06347
- RLHF foundations: Christiano, P. et al. Deep Reinforcement Learning from Human Preferences. NeurIPS 2017. https://arxiv.org/abs/1706.03741
- Alignment framework: Askell, A. et al. A General Language Assistant as a Laboratory for Alignment. 2021. https://arxiv.org/abs/2112.00861