Chinchilla: Why Bigger Models Are Not Always Better

Mon, 21 Sep 2026 00:00:00 +0000

Introduction: The Parameter Race Missed Half the Equation

The large-model race of 2020–2021 had an easy scoreboard: more accelerators should produce more parameters. GPT-3 had 175B parameters, Gopher had 280B, and Megatron-Turing NLG reached 530B. Parameter count kept rising while training datasets often remained near the 300B-token scale.

That strategy was not irrational. Kaplan et al.’s 2020 scaling laws suggested that larger models were more sample-efficient. Under a fixed compute budget, the apparent optimum was to spend aggressively on model size and stop training well before convergence. But would the same allocation remain optimal across wider model sizes and longer training runs?

DeepMind’s 2022 paper, Training Compute-Optimal Large Language Models, gave a striking answer: for their available compute, the leading models of the time were generally too large and trained on too little data. A smaller model that reads much more data can be better than a giant, undertrained one.

Cover: model capacity and training data must share the same compute budget

The researchers trained more than 400 models ranging from 70M to over 16B parameters and from 5B to 500B training tokens. They then ran the decisive test: using essentially Gopher’s training FLOPs, they trained Chinchilla with only 70B parameters but 1.4T tokens. The model was four times smaller than Gopher and nevertheless won across almost every measured task.

Chinchilla therefore matters less as a particular model than as a reframing of the design question. Instead of asking “How large can we make the model?”, it asks: Given a compute budget, how should we divide it between parameters and data?

The Compute Budget: What Are We Actually Spending?

For a standard dense Transformer, dominant training compute can be approximated as:

$$C \approx 6ND$$

Here, $C$ is training FLOPs, $N$ is the number of non-embedding parameters, and $D$ is the number of training tokens. The constant six approximates forward and backward computation. Real engineering costs vary with sequence length, activation recomputation, and implementation details, but this expression captures the scaling trade-off.

With $C$ fixed, $N$ and $D$ sit on a seesaw. Doubling parameters requires halving tokens. The goal is not to maximize either variable; it is to minimize the final validation loss.

The paper expresses the two bottlenecks with a separable empirical loss model:

$$L(N,D)=E+\frac{A}{N^\alpha}+\frac{B}{D^\beta}$$

$E$ is the irreducible entropy of the data distribution.
$A/N^\alpha$ captures loss from insufficient model capacity.
$B/D^\beta$ captures loss from insufficient data and optimization.

A very large model does not make the data-limited term disappear. In Chinchilla’s context, an “undertrained” model is not merely one whose optimizer stopped early. It is a model that has seen too few tokens relative to its capacity.

Three Methods, One Conclusion

The paper did not trust a single fitted curve. It estimated the compute-optimal frontier in three independent ways.

The first method fixed model size, varied training-token count, and used the learning curves to identify the lowest attainable loss at each compute budget. It then fit optimal model and data sizes as functions of compute:

$$N_{opt}\propto C^{0.50},\qquad D_{opt}\propto C^{0.50}$$

The second method created IsoFLOP profiles. For each fixed FLOP budget, the team trained several combinations of parameter count and token count. Plotting final loss against model size produces a visible valley; the bottom of that valley is the optimum for that budget. Fitting those minima produced exponents near 0.49 for parameters and 0.51 for tokens.

The third method fit the parametric loss function directly to all final-loss observations. Its exponents, approximately 0.46 and 0.54, were not identical, but the direction was the same. Across all three approaches, increased compute should be allocated in roughly equal proportions to model size and training data.

This corrected the earlier Kaplan prescription, which directed most additional compute toward a larger model. Engineers often summarize Chinchilla as “about 20 tokens per parameter”: 70B × 20 is roughly 1.4T. That ratio is a useful landmark, not a law of nature. It depends on the model family, data distribution, objective, and fitted range. The durable result is joint scaling, not blind adherence to 20:1.

Same Compute: How 70B Beat 280B

Figure 1: Gopher and Chinchilla use similar training compute; Chinchilla reallocates the budget from parameters to tokens

Gopher has 280B parameters and was trained on about 300B tokens. Chinchilla has 70B parameters and was trained on 1.4T tokens. Roughly speaking, the model became four times smaller while the dataset became about four times larger, leaving the product $N\times D$ in the same range.

This was not a win produced by a revolutionary architecture. The paper says Chinchilla largely retained Gopher’s architecture and training setup, with differences that included the data mixture, AdamW, and a slightly modified SentencePiece tokenizer. The central evidence is about compute allocation, not an architectural trick.

The results were broad:

Metric	Gopher 280B	Chinchilla 70B	Interpretation
Training tokens	about 300B	1.4T	More complete use of model capacity
MMLU 5-shot	about 60.0%	67.6%	A 7.6-point average gain
LAMBADA	74.5%	77.4%	Better prediction with a smaller model
BIG-bench average	54.4%	65.1%	Behind on only 4 of 62 tasks

On MMLU, Chinchilla beat Gopher on 51 of 57 tasks, tied on two, and lost on only four. It also outperformed GPT-3, Jurassic-1, and 530B MT-NLG across a wide range of evaluations. The most important finding is not a particular benchmark decimal. It is the counterfactual: spending the same compute on excessive parameter count can produce a worse model.

Training-Optimal Is Not Product-Optimal

Chinchilla optimizes a narrow objective: minimize training loss for a fixed pretraining FLOP budget. A deployed product has another large cost center—serving the model.

In this experiment, Chinchilla improved both quality and deployment economics. A 70B model needs less memory, bandwidth, and inference compute than a 280B model. More generally, however, “train once” and “serve a billion times” belong in the same lifetime budget.

The 2024 paper Beyond Chinchilla-Optimal adds expected inference demand to the optimization. Its conclusion is that heavily used models may rationally be smaller than the Chinchilla training optimum and trained for longer. Extra one-time pretraining compute can be exchanged for lower cost on every future request.

Meta’s official Llama 3 report illustrates this shift. Meta says the Chinchilla-optimal training amount for an 8B model is around 200B tokens, yet both its 8B and 70B models continued improving when trained on as many as 15T tokens. This does not refute Chinchilla. The objective changed from lowest loss at fixed training FLOPs to a product trade-off that includes quality and repeated inference efficiency.

A practical decision process therefore has three layers:

Estimate parameter-data-loss relationships with scaling experiments.
Add constraints from data quality, hardware, and the training window.
Include expected lifetime traffic, latency, memory, and serving cost.

The 20-token heuristic is a starting point, not a purchase order.

Data Becomes the Scarce Resource

Before Chinchilla, the obvious bottleneck was distributing a larger model over more accelerators. After Chinchilla, data engineering became equally central. The paper estimated that a 175B model on its compute-optimal frontier would need several trillion tokens rather than GPT-3’s roughly 300B. At larger scales, the demand for high-quality, deduplicated, legally usable data grows rapidly.

“More tokens” cannot simply mean crawling and repeating more web pages. Poor data changes the constants and exponents in the loss relationship. Repetition can encourage memorization rather than generalization. Deduplication, filtering, language and domain balance, provenance, and contamination checks determine whether an additional token contributes new information.

To me, Chinchilla’s deepest industrial effect was to turn data from a pipeline input into a first-class model-design variable. An architecture team should not pick a parameter count first and ask the data team to fill it later. Capacity and corpus have to be designed under one budget.

Guardrail Perspective

Scaling laws look like efficiency research, but they directly shape safety boundaries.

First, compute-optimal is not risk-optimal. More high-quality tokens increase coverage and generalization, but they can also improve capabilities in cyber operations, persuasion, or other sensitive domains. Capability evaluations and dangerous-capability evaluations must scale with pretraining, not arrive as an afterthought.

Second, pressure for trillions of tokens creates provenance and privacy hazards. Teams may be tempted to relax licensing, personal-data removal, or traceability standards. A corpus that is compute-optimal can still be unacceptable on copyright, privacy, bias, or consent grounds. Data governance belongs inside the optimization constraints.

Third, a scaling law is an extrapolation tool, not a safety proof. The fits come from particular architectures, mixtures, and compute ranges. Extrapolating several orders of magnitude carries uncertainty, and predictable loss does not imply predictable behavior. Deployment still requires empirical evaluations, red teaming, tiered access, and monitoring.

My conclusion is that Chinchilla taught the industry not to waste training compute. Safety practice adds a necessary second sentence: do not consume governance and evaluation margins merely to satisfy a scaling curve. A useful optimum must account for engineering, economics, and risk together.

Conclusion

Chinchilla ended the simplistic parameter-count race with an unusually clean comparison: a 70B model trained on 1.4T tokens beat 280B Gopher at roughly the same training compute.

Three principles remain useful:

Jointly optimize parameters and data under a fixed training budget.
Scaling parameter count and token count in roughly equal proportions is more efficient than scaling parameters alone.
Training optimality is only one objective; data quality and lifetime inference economics can favor smaller models trained for longer.

Parameter count describes how much capacity a model contains. Token count determines whether that capacity is actually trained. After Chinchilla, model “scale” is no longer one number—it is a system formed by parameters, data, compute, and deployment demand.

References

Hoffmann, J. et al. Training Compute-Optimal Large Language Models. NeurIPS 2022. https://arxiv.org/abs/2203.15556
Google DeepMind. An empirical analysis of compute-optimal large language model training. https://deepmind.google/blog/an-empirical-analysis-of-compute-optimal-large-language-model-training/
Kaplan, J. et al. Scaling Laws for Neural Language Models. 2020. https://arxiv.org/abs/2001.08361
OpenAI. Scaling laws for neural language models. https://openai.com/index/scaling-laws-for-neural-language-models/
Sardana, N. et al. Beyond Chinchilla-Optimal: Accounting for Inference in Language Model Scaling Laws. ICML 2024. https://proceedings.mlr.press/v235/sardana24a.html
Meta AI. Introducing Meta Llama 3. https://ai.meta.com/blog/meta-llama-3/

Scaling Laws on Lee