DeepSeek-V2/V3: The Two Bills Behind MLA and MoE

Wed, 23 Sep 2026 00:00:00 +0000

Writing “671B parameters” next to a model invites a misleading picture: every generated token must somehow move and compute all 671 billion parameters. Writing “128K context” creates a second simplification, as if the only concern were the arithmetic complexity of attention. The most instructive feature of DeepSeek-V2 and V3 is that they split those costs into two different bills: how much history each token must preserve, and how much model capacity each token must activate.

Multi-head Latent Attention (MLA) addresses the first bill. Instead of keeping complete keys and values for every attention head, it stores a low-dimensional joint latent from which their content can be recovered. DeepSeekMoE addresses the second. The model may own many expert parameters, while the router sends a token through only a few of them. Together they express a systems principle: total capacity, per-token compute, and serving memory do not have to scale together.

That principle matters more than another claim to model size. The DeepSeek-V2 paper reports a 93.3% reduction in KV cache and up to 5.76 times maximum generation throughput relative to DeepSeek 67B, while using 236B total parameters, 21B activated per token, and a 128K context window. V3 retained MLA and DeepSeekMoE, then scaled to 671B total and roughly 37B activated parameters. These are measurements from the respective reports and should not be exported to unrelated hardware, precision, or serving settings. They do establish that the design survived beyond a small ablation model.

The First Bill: What Generation Must Remember

Autoregressive generation adds one token at a time. To avoid recomputing the entire prefix at every step, a Transformer retains the previous keys and values in a KV cache. Standard multi-head attention (MHA) stores this many elements per layer and token:

$$2n_h d_h$$

Here $n_h$ is the number of attention heads and $d_h$ is the dimension of one head; the factor two accounts for K and V. Multiplying by layer count, sequence length, batch size, and bytes per element gives the total memory. The cache grows linearly with context and concurrency, and every decoding step reads an ever-longer history. Capacity and memory bandwidth can therefore matter more than peak matrix-multiplication throughput.

Multi-query attention shares one K/V head across all query heads. Grouped-query attention shares several. Both shrink the cache, but they also force query heads to share their historical representation. DeepSeek-V2 asked a more aggressive question than “How few KV heads can we keep?”: Why cache expanded K and V at all?

Figure 1: MHA caches complete K/V projections for every head; MLA caches a joint K/V latent plus a separate positional key

MLA first down-projects the current hidden state $h_t$ into a compact latent:

$$c_t^{KV}=W^{DKV}h_t$$

Two up-projections can then recover content keys and values:

$$k_t^C=W^{UK}c_t^{KV},\qquad v_t^C=W^{UV}c_t^{KV}$$

During training, different heads can still learn different K/V representations. During generation, only $c_t^{KV}$ must remain. The important word is not merely “low-rank” but joint: the same latent serves both keys and values, and its dimension $d_c$ is much smaller than the fully expanded $2n_hd_h$.

The more subtle step changes the computation graph. $W^{UK}$ can be absorbed into the query projection, while $W^{UV}$ can be absorbed into the output projection. Attention scores can therefore be computed between transformed queries and cached latents, and value aggregation can happen in latent space before the final mapping. DeepSeek’s reference V3 code exposes the idea as two literal implementations named naive and absorb. The first materializes and caches full K/V tensors. The second caches the normalized low-rank KV and positional component. The code makes the paper’s algebra inspectable as an execution path.

RoPE Is the Operation That Cannot Simply Move

If the chain contained only linear projections, absorption would be straightforward. Rotary position embedding complicates it because the rotation applied to queries and keys depends on token position. If RoPE is applied after reconstructing $k_t^C$, a position-dependent matrix sits between the query-side projection and $W^{UK}$. Those matrices do not commute. The up-projection can no longer be absorbed once and reused; recovering position-sensitive keys for the prefix would erase the practical benefit of compression.

DeepSeek’s solution separates query and key channels into content and position. The content path uses low-rank compression and permits projection absorption. A small additional query/key branch carries RoPE. Each head has its positional query $q_{t,i}^{R}$, while the positional key $k_t^{R}$ is shared across heads. The final score uses the concatenated content and positional components, and the cache retains only:

$$c_t^{KV}\quad\text{and}\quad k_t^R$$

Figure 2: decoupled RoPE moves the position-dependent operation outside the absorbable content projection path

DeepSeek-V2 sets $d_c=4d_h$ and $d_h^R=d_h/2$, so the cache contains $(d_c+d_h^R)=4.5d_h$ elements per layer and token. The paper describes that as roughly equivalent to GQA with only 2.25 groups, while its ablations retain performance comparable to or better than MHA. “Low-rank” should not be read as free compression. Representation capacity, training stability, and specialized kernels all require careful design. MLA reduces the state that must remain resident and move repeatedly; it does not eliminate every operation in every phase.

The Second Bill: Many Parameters Need Not All Compute

After attention, feed-forward networks account for another large fraction of Transformer work. Every token in a dense model passes through the same FFN weights, tying model capacity closely to compute. A mixture of experts replaces one FFN with many experts and uses a router to select a top-K subset. Total parameters can grow quickly while active parameters per token grow much more slowly.

DeepSeekMoE modifies conventional top-K MoE in two deliberate ways. First, it divides large experts into more fine-grained ones and selects more of those smaller units. With approximately the same active parameter budget, the router can form a richer set of combinations. Second, it isolates some shared experts through which every token passes. They learn common knowledge, reducing the need for routed experts to duplicate the same foundation.

Figure 3: shared experts carry common capability while fine-grained routed experts provide specialized capacity in combinations

This is why total and active parameters must be reported together. V2’s 236B/21B and V3’s 671B/37B are not interchangeable marketing numbers. Total parameters approximate the capacity and storage footprint of the weights; active parameters more closely describe the weights participating in one forward pass. Active parameters are not a complete latency formula, however. Expert weights must reach compute units, tokens cross devices, and an overloaded expert can stall the batch. MoE converts part of the compute problem into communication, scheduling, and load balancing.

Why V3 Stopped Making Balance a Direct Model Penalty

Earlier MoE systems commonly added an auxiliary training loss that pushed token assignments toward a more even expert distribution. It prevents a few experts from overloading, but it also mixes a systems objective into language modeling. One domain may genuinely benefit from concentrating on a particular group of experts, while a sequence-level balance loss tells each sequence to spread out.

One of V3’s main changes is an expert bias $b_i$ used for top-K selection. Routing still begins with token-expert affinity scores, but a dynamic bias is added to the selection score. An expert that has recently been overloaded receives a lower bias; an underused expert receives a higher one. Crucially, the bias affects which experts are selected without directly scaling their outputs or adding a main load-balancing term to the language-model loss. V3 retains a very small sequence-wise auxiliary loss to prevent extreme imbalance inside one sequence. “Auxiliary-loss-free” is therefore best read as the primary expert load-balancing mechanism does not depend on an auxiliary loss.

V3’s ablations support a bounded claim. At 15.7B and 228.7B baseline scales, the method outperformed the purely auxiliary-loss-based alternative on most reported benchmarks. A further batch-wise versus sequence-wise comparison suggested that looser batch-level balancing permits clearer domain specialization. It does not prove that routing bias is universally best for every batch size, network, or traffic distribution. The deployment design still uses redundant experts and dynamic scheduling to absorb skew in real requests.

End-to-End Efficiency Requires Both Bills

MLA and MoE are often taught as separate modules, yet they address adjacent serving bottlenecks. MLA makes the per-request historical state smaller. MoE lets each new token use only a fraction of total model capacity. The first can make room for a longer context or a larger batch; the second lets capacity grow without proportional token compute. With only one of them, the other bill can remain the limiting factor.

This is also why an architecture paper cannot end at the equations. MLA needs an attention kernel that consumes the compressed cache directly. MoE needs efficient all-to-all communication, expert parallelism, and load-aware scheduling. DeepSeek later released FlashMLA as dedicated MLA decoding kernels, while the V3 report devotes substantial space to communication overlap and cross-node routing. Architecture reduces the amount that must theoretically move or activate; systems software decides whether the reduction becomes throughput.

The durable lesson is not that low-rank attention always beats GQA or that MoE always beats dense models. MLA needs a suitable compression dimension, a compatible positional path, and kernel support. MoE benefits from enough load and capable interconnects; small batches may instead expose routing overhead and latency. The safer principle is: do not ask only how many parameters a model owns or how long its context is. Ask what each token stores, reads, activates, and sends across the hardware. DeepSeek-V2 and V3 made those four questions part of the architecture itself.

References

DeepSeek-AI, DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model, 2024.
DeepSeek-AI, DeepSeek-V3 Technical Report, 2024/2025.
Dai et al., DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models, 2024.
DeepSeek-AI, DeepSeek-V3 official repository and reference inference implementation.
DeepSeek-AI, DeepSeek-V2 official repository.
DeepSeek-AI, FlashMLA: Efficient Multi-head Latent Attention Kernels.

KV Cache on Lee