Transformer on Lee

GPT-1/2: Generative Pretraining and the Seeds of Few-Shot

Fri, 28 Aug 2026 00:00:00 +0000

From BERT to GPT: Two Symmetric Bets

2018 split NLP into two lines. BERT went bidirectional-encoder for understanding. GPT-1 went decoder-only for generation. Same Transformer substrate, opposite architecture configurations — a fork that defined the next five years.

Core Design: Decoder-Only and Autoregression

GPT-1 extracts the Transformer decoder, drops cross-attention, stacks 12 layers of masked self-attention + FFN.

Masked self-attention — each token sees only itself and the left context. Future scores are set to -∞ before softmax, so their weights become 0:

scores = Q @ K.T / sqrt(d_k)
mask = torch.triu(torch.ones(L, L), diagonal=1).bool()
scores[mask] = -float('inf')
attn = softmax(scores, dim=-1)

Pre-norm — GPT-1 places LayerNorm before each sublayer (post-norm in the original Transformer). This small change makes deep stacks trainable.

Training Details

	GPT-1	GPT-2 Small
Layers	12	12
Hidden dim	768	768
Heads	12	12
Params	117M	124M
Data	BookCorpus (~1B words)	WebText (~40GB)
Batch	64	512
Sequence length	512	1024

Why Decoder-Only Can Understand

The counterintuitive claim: predicting the next token requires understanding everything before it — syntax, coreference, commonsense. If the model predicts “groceries” after “Xiaoming went to the supermarket and bought some ___”, it must resolve what supermarkets sell.

GPT-1 hit SOTA on 9 of 12 NLP tasks — behind BERT on absolute scores, but proof that generation subsumes understanding.

GPT-2: Implicit Multitask Learning

GPT-2 scaled up (1.5B max) and found something more important: zero-shot task transfer.

Machine translation (WMT-14 En→Fr): zero-shot BLEU 11.5 — no parallel data ever seen
Reading comprehension (CoQA): 55 F1 zero-shot
Summarization (CNN/DM): competitive ROUGE-L without fine-tuning

The model learned multiple tasks implicitly from language modeling alone.

Scaling Without Saturation

Model	Params	LAMBADA (zero-shot)
GPT-2 Small	124M	45.0%
GPT-2 Medium	355M	55.0%
GPT-2 Large	774M	58.0%
GPT-2 XL	1.5B	63.0%

No plateau. This curve directly motivated GPT-3’s 175B and the scaling-laws program.

Why GPT-1/2 Matters

Decoder-only won — GPT-3, LLaMA, Mistral, DeepSeek all follow this line
Zero-shot changed everything — GPT-2’s discovery that models transfer without fine-tuning led directly to in-context learning, instruction tuning, and RLHF
Scaling as strategy — 117M → 1.5B → 175B was not accidental; GPT-2’s data justified it

References

Radford, A., et al. (2018). Improving Language Understanding by Generative Pre-Training. OpenAI.
Radford, A., et al. (2019). Language Models are Unsupervised Multitask Learners. OpenAI.
The Illustrated GPT-2: http://jalammar.github.io/illustrated-gpt2/

BERT: What Exactly Is 'Bidirectional'?

Thu, 27 Aug 2026 00:00:00 +0000

Prologue: GPT-1’s Unidirectional Constraint

GPT-1 uses a Transformer decoder as an autoregressive LM: each token attends only to itself and previous tokens. Reading a sentence becomes “forward-only”. For “Xiaoming gave Xiaohong ___ a book”, GPT cannot see the object “book” that follows the blank.

BERT’s name — Bidirectional Encoder Representations from Transformers — is about fixing exactly this.

But “bidirectional” costs almost nothing architecturally: the only change is the attention mask. The hard part is training it without label leakage.

1. Bidirectionality Lives in the Attention Mask

GPT uses a lower-triangular causal mask. BERT uses a full matrix — every token attends to every token, including itself and future positions. No new parameters, no layer changes. Just a mask of ones instead of a triangle.

2. Why You Can’t Use a Standard LM Objective

If the model can see token t when predicting token t, the optimal solution is to copy the input. The LM objective degenerates into transcription. BERT’s solution: Masked Language Modeling.

The 15% Masking Rate

The paper states: “we mask 15% of all WordPiece tokens in each sequence at random.” No ablation over mask rates exists in the paper — 15% is a heuristic, not an optimized value.

The 80/10/10 Split

Selected positions are replaced:

80% → [MASK] token
10% → random token
10% → unchanged

The 10% unchanged keeps the model honest: it must maintain a distributional representation for every input token, because it never knows which positions will be queried.

3. NSP: The Task That Later Fell

Next Sentence Prediction: 50% of pairs are real adjacent sentences, 50% random. [CLS] output is used for binary classification.

The actual ablation numbers (Table 5 of the paper):

Task	MNLI-m	QNLI	MRPC	SST-2	SQuAD
BERT_BASE	84.4	88.4	86.7	92.7	88.5
No NSP	83.9	84.9	86.5	92.6	87.9

Removing NSP costs 3.5 points on QNLI (a sentence-pair task) but only 0.5 on MNLI and 0.1 on SST-2. The effect is task-dependent.

RoBERTa (2019) later showed that with more data and longer training, removing NSP is neutral or slightly better:

Setup	SQuAD 1.1/2.0	MNLI	RACE
SEGMENT-PAIR + NSP	90.4 / 78.7	84.0	64.2
FULL-SENTENCES, no NSP	90.4 / 79.1	84.7	64.8

ALBERT replaced NSP with SOP (Sentence Order Prediction) — harder and more useful for discourse coherence.

4. Details Often Gotten Wrong

GELU Attribution

BERT uses GELU “following OpenAI GPT” — GPT-1 (June 2018) adopted GELU first. BERT is the inheritor, not the inventor.

SQuAD Fine-tuning Has Parameters

The start/end predictors are linear layers on top of BERT output: start_logits = W_s·h_i + b_s. Claiming “no additional parameters” is wrong.

Pre-training Compute

Batch: 256 sequences × 512 tokens = 128,000 tokens/batch
1M steps ≈ 40 epochs over 3.3B words
90% of steps use seq_len 128 (attention is quadratic); 10% use 512

5. BERT’s Real Contribution

Three layers:

Architecture: full attention instead of causal — near-zero cost, huge representational gain
Training: MLM solves the “seeing yourself” problem — the objective itself prevents leakage
Paradigm: universal pretraining + lightweight fine-tuning replaced task-specific architectures

Limitations

512 max sequence length (learned positional embeddings)
Pretrain/fine-tune mismatch persists ([MASK] never appears downstream)
Cannot generate — encoder-only has no causal factorization
NSP is inefficient: half the training signal is about sentence adjacency, later shown to be replaceable

References

Devlin, J., et al. (2018). BERT: Pre-training of Deep Bidirectional Transformers. arXiv:1810.04805.
Liu, Y., et al. (2019). RoBERTa. arXiv:1907.11692.
Lan, Z., et al. (2020). ALBERT. arXiv:1909.11942.

Transformer: The Starting Point of Everything — What Problem Did It Actually Solve?

Wed, 26 Aug 2026 00:00:00 +0000

Prologue: The Bottleneck of Sequential Processing

Before 2017, sequence models meant RNNs and LSTMs. They process tokens one at a time: hidden state at step t depends on step t-1. This creates two fatal problems:

No parallelism — training cannot be parallelized across time steps, so you can’t leverage GPUs effectively
Long-range information decay — even with gating mechanisms, information from 100 steps ago gets diluted through repeated transformations

Figure 1: RNN sequential processing — hidden state must pass through every step

The Transformer paper’s answer was radical: throw away recurrence entirely. Use attention for everything.

Self-Attention: The Core Mechanism

The formula every LLM engineer knows:

$$\text{Attention}(Q, K, V) = \text{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right)V$$

Every token computes a query, key, and value vector. The attention score between token i and token j is the dot product of query_i and key_j, scaled by √d_k, then softmaxed into weights. The output for token i is a weighted sum of all value vectors.

Why the √d_k scaling? For large d_k, dot products grow large in magnitude, pushing softmax into regions with tiny gradients. Dividing by √d_k keeps the variance stable.

# Scaled dot-product attention
scores = Q @ K.T / math.sqrt(d_k)     # [seq, seq]
attn = softmax(scores, dim=-1)        # weights sum to 1
output = attn @ V                      # [seq, d_v]

The key property: every token can attend to every other token in O(1) sequential steps. Long-range dependencies are now direct paths, not chains of transformations.

Figure 2: Scaled dot-product attention

Multi-Head Attention: Looking from Different Angles

One attention head learns one type of relationship. Multi-head attention runs h heads in parallel (each with smaller d_k), concatenates their outputs:

$$\text{MultiHead}(Q,K,V) = \text{Concat}(\text{head}_1, \ldots, \text{head}_h)W^O$$

where each head_i = Attention(QW_i^Q, KW_i^K, VW_i^V).

With 8 heads of dimension 64 (total 512), one head might track syntax, another coreference, another positional patterns. Empirically, heads specialize without explicit instruction.

Figure 3: Multi-head attention — h parallel attention operations

Positional Encoding: Where Am I?

Attention is permutation-invariant — it has no notion of word order. “Dog bites man” and “man bites dog” produce identical attention patterns. The fix: add positional encodings to input embeddings.

The original paper used sinusoidal functions:

$$PE_{(pos, 2i)} = \sin\left(\frac{pos}{10000^{2i/d}}\right), \quad PE_{(pos, 2i+1)} = \cos\left(\frac{pos}{10000^{2i/d}}\right)$$

Each dimension corresponds to a sinusoid of different wavelength. The intuition: relative positions can be computed as linear transformations, and the model can extrapolate to longer sequences.

Training Details

Parameter	Value
Layers (Encoder/Decoder)	6 / 6
Model dimension	512
Heads	8
FFN inner dimension	2048
Optimizer	Adam (β1=0.9, β2=0.98)
Learning rate schedule	warmup 4000 steps, then ∝ d_model^-0.5 · step^-0.5
Dropout	0.1
Label smoothing	0.1

Why Transformer Won

Full parallelism — all tokens processed simultaneously during training
Direct long-range paths — max path length between any two tokens is O(1)
Scalable — architecture has no recurrence bottleneck, so it scales with compute

This third property turned out to matter most. Transformer became the substrate for BERT, GPT, and every modern LLM — not because attention is theoretically optimal, but because it exploits GPUs perfectly and keeps getting better as you scale.

References

Vaswani, A., et al. (2017). Attention Is All You Need. arXiv:1706.03762.
The Annotated Transformer: http://nlp.seas.harvard.edu/2018/04/03/attention.html