LLM Architecture on Lee

GPT-3: In-Context Learning and the Few-Shot Explosion

Wed, 02 Sep 2026 00:00:00 +0000

GPT-2’s Unfinished Question

GPT-2 showed two things: zero-shot capability exists, and scaling shows no saturation. But at 1.5B, zero-shot CoQA scored 55 F1 while fine-tuned SOTA was 90.7 — a huge gap.

GPT-3’s question: what happens at 100× the scale?

The 175B Configuration

	GPT-2 XL	GPT-3
Layers	48	96
d_model	1600	12288
Heads	25	96
Context	1024	2048
Params	1.5B	175B

Trained on ~300B tokens (Common Crawl + WebText + books + Wikipedia), roughly 570× the text of GPT-2.

In-Context Learning: The Core Discovery

GPT-3’s headline result: few-shot learning via prompts, no gradient updates at all.

Translate English to French:
sea otter => loutre de mer
cheese => fromage
hello =>

The model completes “bonjour” — not because it was fine-tuned on translation, but because attention over the prompt’s examples conditions the output distribution.

The Scaling Pattern

Across tasks, performance follows a consistent pattern: zero-shot < one-shot < few-shot, and the gap widens with model size. Small models barely benefit from examples; 175B models benefit dramatically.

Task	Fine-tuned SOTA	GPT-3 zero-shot	GPT-3 few-shot
SuperGLUE	89.8	52.9	85.6
TriviaQA	—	54.2	71.2
LAMBADA	63.2	63.8	76.2
Arithmetic (2-digit +)	—	26.5	80.4

Few-shot GPT-3 matches or exceeds fine-tuned models on many tasks — with zero parameter updates.

Why ICL Works (Mechanistically)

In-context learning is not fully understood, but the leading hypotheses:

Implicit gradient descent — the forward pass over examples approximates a meta-learned update (von Oswald et al., 2022)
Induction heads — specific attention heads implement “find previous occurrence of current token, copy what followed” (Olsson et al., 2022)
Bayesian inference — the model treats the prompt as evidence and conditions on a latent task distribution (Xie et al., 2021)

None is the full story; all capture parts of it.

The Scaling Laws Connection

Kaplan et al. (2020) showed loss follows power laws in parameters, data, and compute:

$$L(N) \propto N^{-0.076}$$

GPT-3 sits on the predicted curve. Chinchilla (2022) later showed the optimal token/parameter ratio is ~20 tokens per parameter — GPT-3 was undertrained by that measure, which is why LLaMA-65B (1.4T tokens) matches it at ~⅓ the size.

What GPT-3 Changed

Prompting as interface — natural language became the programming interface for models
The API economy — no fine-tuning means one model serves all users
Scaling as business strategy — GPT-3 justified compute investment at a scale no paper had before

Limitations

Hallucination — fluent but factually wrong outputs (the term entered common usage with GPT-3)
No memory — 2048-token context is tiny; later models extended this by orders of magnitude
Compute cost — few-shot inference on 175B was extremely expensive
Undertrained by modern standards — Chinchilla-optimal training would use ~3.5T tokens

References

Brown, T., et al. (2020). Language Models are Few-Shot Learners. arXiv:2005.14165.
Kaplan, J., et al. (2020). Scaling Laws for Neural Language Models. arXiv:2001.08361.
Hoffmann, J., et al. (2022). Training Compute-Optimal Large Language Models. arXiv:2203.15556.

DeepSeek-R1: Eliciting Reasoning in LLMs via Reinforcement Learning

Tue, 01 Sep 2026 00:00:00 +0000

R1-Zero: Pure RL, No SFT

The DeepSeek-R1 paper has two models:

R1-Zero: base model + RL directly, no supervised fine-tuning
R1: SFT cold start → RL → rejection sampling → more SFT → more RL

R1-Zero is the scientifically interesting one: applied directly to DeepSeek-V3-Base with GRPO, it develops reasoning behaviors — self-verification, reflection, long chains of thought — without any demonstration data.

GRPO: Group Relative Policy Optimization

PPO needs a separate value model (about one extra model’s memory). GRPO drops it: sample G outputs per prompt from the old policy, use the group’s mean as baseline:

$$A_i = \frac{r_i - \text{mean}(r_1, \ldots, r_G)}{\text{std}(r_1, \ldots, r_G)}$$

This turns the advantage estimate into a group-relative comparison, removing the critic network entirely.

Rule-Based Rewards

For math/code, rewards are verifiable:

Correctness: exact-match or unit tests pass
Format: response contains ``` reasoning tags and answer tags

No learned reward model → no reward hacking against a proxy. The model cannot fool a unit test.

Figure 1: R1 training pipeline — cold start, RL, rejection sampling, second RL round

Emergent Behaviors in R1-Zero

Training curves show the “aha moment”: around a certain RL step, response length jumps and accuracy climbs discontinuously. Behaviors that appear without being rewarded:

Self-verification: “Wait, let me check this again…”
Backtracking: abandoning a failed approach mid-solution
Reflection: explicitly stating assumptions and testing them

The paper’s interpretation: given only correctness rewards and enough exploration, chain-of-thought is the most natural path to correct answers.

Why R1 (the Two-Stage Version) Is Still Needed

R1-Zero has problems: non-English mixing, endless repetition, weak instruction following. The full R1 pipeline fixes these:

SFT cold start (thousands of long-CoT examples) — gives the model a usable format
Reasoning RL — GRPO on verifiable tasks
Rejection sampling + SFT — generate diverse reasoning data, filter by correctness, retrain
Second RL round — now including human-preference rewards for helpfulness and harmlessness

Results

Benchmark	DeepSeek-V3	R1-Zero	R1
AIME 2024	39.2	71.0	79.8
MATH-500	90.2	95.9	97.3
Codeforces (percentile)	58.7	—	96.3
GPQA Diamond	59.1	—	71.5

R1 matches OpenAI o1-1217 on AIME and MATH.

Why This Matters

Reasoning is learnable by RL — no process-supervision data (as o1 likely used) required, outcome rewards suffice
Open replication — the recipe, code, and weights are all open, unlike o1
The efficiency question — R1 achieves o1-level results with a much smaller infrastructure budget, challenging the “scale is everything” narrative

Limitations

Language mixing persists in long reasoning chains
Sensitive to prompt format (the paper itself notes format brittleness)
RL for general domains (writing, open-ended QA) still needs preference models, where reward hacking returns

References

DeepSeek-AI (2025). DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning. arXiv:2501.12948.
Shao, Z., et al. (2024). DeepSeekMath: Pushing the Limits of Mathematical Reasoning (GRPO origin). arXiv:2402.03300.

GPT-1/2: Generative Pretraining and the Seeds of Few-Shot

Fri, 28 Aug 2026 00:00:00 +0000

From BERT to GPT: Two Symmetric Bets

2018 split NLP into two lines. BERT went bidirectional-encoder for understanding. GPT-1 went decoder-only for generation. Same Transformer substrate, opposite architecture configurations — a fork that defined the next five years.

Core Design: Decoder-Only and Autoregression

GPT-1 extracts the Transformer decoder, drops cross-attention, stacks 12 layers of masked self-attention + FFN.

Masked self-attention — each token sees only itself and the left context. Future scores are set to -∞ before softmax, so their weights become 0:

scores = Q @ K.T / sqrt(d_k)
mask = torch.triu(torch.ones(L, L), diagonal=1).bool()
scores[mask] = -float('inf')
attn = softmax(scores, dim=-1)

Pre-norm — GPT-1 places LayerNorm before each sublayer (post-norm in the original Transformer). This small change makes deep stacks trainable.

Training Details

	GPT-1	GPT-2 Small
Layers	12	12
Hidden dim	768	768
Heads	12	12
Params	117M	124M
Data	BookCorpus (~1B words)	WebText (~40GB)
Batch	64	512
Sequence length	512	1024

Why Decoder-Only Can Understand

The counterintuitive claim: predicting the next token requires understanding everything before it — syntax, coreference, commonsense. If the model predicts “groceries” after “Xiaoming went to the supermarket and bought some ___”, it must resolve what supermarkets sell.

GPT-1 hit SOTA on 9 of 12 NLP tasks — behind BERT on absolute scores, but proof that generation subsumes understanding.

GPT-2: Implicit Multitask Learning

GPT-2 scaled up (1.5B max) and found something more important: zero-shot task transfer.

Machine translation (WMT-14 En→Fr): zero-shot BLEU 11.5 — no parallel data ever seen
Reading comprehension (CoQA): 55 F1 zero-shot
Summarization (CNN/DM): competitive ROUGE-L without fine-tuning

The model learned multiple tasks implicitly from language modeling alone.

Scaling Without Saturation

Model	Params	LAMBADA (zero-shot)
GPT-2 Small	124M	45.0%
GPT-2 Medium	355M	55.0%
GPT-2 Large	774M	58.0%
GPT-2 XL	1.5B	63.0%

No plateau. This curve directly motivated GPT-3’s 175B and the scaling-laws program.

Why GPT-1/2 Matters

Decoder-only won — GPT-3, LLaMA, Mistral, DeepSeek all follow this line
Zero-shot changed everything — GPT-2’s discovery that models transfer without fine-tuning led directly to in-context learning, instruction tuning, and RLHF
Scaling as strategy — 117M → 1.5B → 175B was not accidental; GPT-2’s data justified it

References

Radford, A., et al. (2018). Improving Language Understanding by Generative Pre-Training. OpenAI.
Radford, A., et al. (2019). Language Models are Unsupervised Multitask Learners. OpenAI.
The Illustrated GPT-2: http://jalammar.github.io/illustrated-gpt2/

BERT: What Exactly Is 'Bidirectional'?

Thu, 27 Aug 2026 00:00:00 +0000

Prologue: GPT-1’s Unidirectional Constraint

GPT-1 uses a Transformer decoder as an autoregressive LM: each token attends only to itself and previous tokens. Reading a sentence becomes “forward-only”. For “Xiaoming gave Xiaohong ___ a book”, GPT cannot see the object “book” that follows the blank.

BERT’s name — Bidirectional Encoder Representations from Transformers — is about fixing exactly this.

But “bidirectional” costs almost nothing architecturally: the only change is the attention mask. The hard part is training it without label leakage.

1. Bidirectionality Lives in the Attention Mask

GPT uses a lower-triangular causal mask. BERT uses a full matrix — every token attends to every token, including itself and future positions. No new parameters, no layer changes. Just a mask of ones instead of a triangle.

2. Why You Can’t Use a Standard LM Objective

If the model can see token t when predicting token t, the optimal solution is to copy the input. The LM objective degenerates into transcription. BERT’s solution: Masked Language Modeling.

The 15% Masking Rate

The paper states: “we mask 15% of all WordPiece tokens in each sequence at random.” No ablation over mask rates exists in the paper — 15% is a heuristic, not an optimized value.

The 80/10/10 Split

Selected positions are replaced:

80% → [MASK] token
10% → random token
10% → unchanged

The 10% unchanged keeps the model honest: it must maintain a distributional representation for every input token, because it never knows which positions will be queried.

3. NSP: The Task That Later Fell

Next Sentence Prediction: 50% of pairs are real adjacent sentences, 50% random. [CLS] output is used for binary classification.

The actual ablation numbers (Table 5 of the paper):

Task	MNLI-m	QNLI	MRPC	SST-2	SQuAD
BERT_BASE	84.4	88.4	86.7	92.7	88.5
No NSP	83.9	84.9	86.5	92.6	87.9

Removing NSP costs 3.5 points on QNLI (a sentence-pair task) but only 0.5 on MNLI and 0.1 on SST-2. The effect is task-dependent.

RoBERTa (2019) later showed that with more data and longer training, removing NSP is neutral or slightly better:

Setup	SQuAD 1.1/2.0	MNLI	RACE
SEGMENT-PAIR + NSP	90.4 / 78.7	84.0	64.2
FULL-SENTENCES, no NSP	90.4 / 79.1	84.7	64.8

ALBERT replaced NSP with SOP (Sentence Order Prediction) — harder and more useful for discourse coherence.

4. Details Often Gotten Wrong

GELU Attribution

BERT uses GELU “following OpenAI GPT” — GPT-1 (June 2018) adopted GELU first. BERT is the inheritor, not the inventor.

SQuAD Fine-tuning Has Parameters

The start/end predictors are linear layers on top of BERT output: start_logits = W_s·h_i + b_s. Claiming “no additional parameters” is wrong.

Pre-training Compute

Batch: 256 sequences × 512 tokens = 128,000 tokens/batch
1M steps ≈ 40 epochs over 3.3B words
90% of steps use seq_len 128 (attention is quadratic); 10% use 512

5. BERT’s Real Contribution

Three layers:

Architecture: full attention instead of causal — near-zero cost, huge representational gain
Training: MLM solves the “seeing yourself” problem — the objective itself prevents leakage
Paradigm: universal pretraining + lightweight fine-tuning replaced task-specific architectures

Limitations

512 max sequence length (learned positional embeddings)
Pretrain/fine-tune mismatch persists ([MASK] never appears downstream)
Cannot generate — encoder-only has no causal factorization
NSP is inefficient: half the training signal is about sentence adjacency, later shown to be replaceable

References

Devlin, J., et al. (2018). BERT: Pre-training of Deep Bidirectional Transformers. arXiv:1810.04805.
Liu, Y., et al. (2019). RoBERTa. arXiv:1907.11692.
Lan, Z., et al. (2020). ALBERT. arXiv:1909.11942.

Transformer: The Starting Point of Everything — What Problem Did It Actually Solve?

Wed, 26 Aug 2026 00:00:00 +0000

Prologue: The Bottleneck of Sequential Processing

Before 2017, sequence models meant RNNs and LSTMs. They process tokens one at a time: hidden state at step t depends on step t-1. This creates two fatal problems:

No parallelism — training cannot be parallelized across time steps, so you can’t leverage GPUs effectively
Long-range information decay — even with gating mechanisms, information from 100 steps ago gets diluted through repeated transformations

Figure 1: RNN sequential processing — hidden state must pass through every step

The Transformer paper’s answer was radical: throw away recurrence entirely. Use attention for everything.

Self-Attention: The Core Mechanism

The formula every LLM engineer knows:

$$\text{Attention}(Q, K, V) = \text{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right)V$$

Every token computes a query, key, and value vector. The attention score between token i and token j is the dot product of query_i and key_j, scaled by √d_k, then softmaxed into weights. The output for token i is a weighted sum of all value vectors.

Why the √d_k scaling? For large d_k, dot products grow large in magnitude, pushing softmax into regions with tiny gradients. Dividing by √d_k keeps the variance stable.

# Scaled dot-product attention
scores = Q @ K.T / math.sqrt(d_k)     # [seq, seq]
attn = softmax(scores, dim=-1)        # weights sum to 1
output = attn @ V                      # [seq, d_v]

The key property: every token can attend to every other token in O(1) sequential steps. Long-range dependencies are now direct paths, not chains of transformations.

Figure 2: Scaled dot-product attention

Multi-Head Attention: Looking from Different Angles

One attention head learns one type of relationship. Multi-head attention runs h heads in parallel (each with smaller d_k), concatenates their outputs:

$$\text{MultiHead}(Q,K,V) = \text{Concat}(\text{head}_1, \ldots, \text{head}_h)W^O$$

where each head_i = Attention(QW_i^Q, KW_i^K, VW_i^V).

With 8 heads of dimension 64 (total 512), one head might track syntax, another coreference, another positional patterns. Empirically, heads specialize without explicit instruction.

Figure 3: Multi-head attention — h parallel attention operations

Positional Encoding: Where Am I?

Attention is permutation-invariant — it has no notion of word order. “Dog bites man” and “man bites dog” produce identical attention patterns. The fix: add positional encodings to input embeddings.

The original paper used sinusoidal functions:

$$PE_{(pos, 2i)} = \sin\left(\frac{pos}{10000^{2i/d}}\right), \quad PE_{(pos, 2i+1)} = \cos\left(\frac{pos}{10000^{2i/d}}\right)$$

Each dimension corresponds to a sinusoid of different wavelength. The intuition: relative positions can be computed as linear transformations, and the model can extrapolate to longer sequences.

Training Details

Parameter	Value
Layers (Encoder/Decoder)	6 / 6
Model dimension	512
Heads	8
FFN inner dimension	2048
Optimizer	Adam (β1=0.9, β2=0.98)
Learning rate schedule	warmup 4000 steps, then ∝ d_model^-0.5 · step^-0.5
Dropout	0.1
Label smoothing	0.1

Why Transformer Won

Full parallelism — all tokens processed simultaneously during training
Direct long-range paths — max path length between any two tokens is O(1)
Scalable — architecture has no recurrence bottleneck, so it scales with compute

This third property turned out to matter most. Transformer became the substrate for BERT, GPT, and every modern LLM — not because attention is theoretically optimal, but because it exploits GPUs perfectly and keeps getting better as you scale.

References

Vaswani, A., et al. (2017). Attention Is All You Need. arXiv:1706.03762.
The Annotated Transformer: http://nlp.seas.harvard.edu/2018/04/03/attention.html