FlashAttention neither approximates attention nor removes its quadratic arithmetic. It uses tiling, online softmax, and recomputation to avoid materializing the attention matrix. This article explains the GPU memory hierarchy behind the speedup and the evolution through FA2 and FA3.
Mamba replaces a growing KV cache with a fixed recurrent state. This article explains selective state spaces, hardware-aware scans, and the empirical limits that make hybrid models compelling.
DeepSeek-V2/V3 compress the KV cache and sparsely activate experts. This article explains MLA, decoupled RoPE, DeepSeekMoE, and V3's auxiliary-loss-free load balancing from the data an inference system actually moves.
LLaMA introduced no new Transformer core, yet fuller training, proven components such as RoPE, and deployable sizes reshaped the open-model ecosystem. This article explains the recipe and where GQA actually entered the family.
With the same training compute, a 70B Chinchilla trained on 1.4T tokens beat the 280B Gopher. This article explains compute-optimal scaling, the paper's three fitting methods, and its lasting impact.
