Memory and attention

Batching is the biggest effect on the expert stack, but it is not the only one, and past about eight thousand tokens of context it is not even the largest term.

1. Expert sparsity does not shrink attention. Sparsity applies to the expert layers. Attention weights are read in full every step, and so is the KV cache unless a model's attention selects among its tokens, and at long context they dominate: on published cost decompositions, attention is already several times the expert cost at 8,000 tokens and roughly ten times it at 32,000. That caps the best possible speedup at about 6× at 8K and under 3× at 32K — before any inefficiency at all. Step-3, decoding cost decomposed by model and accelerator ↗
2. Each expert gets a tiny, ragged slice of the batch. At one token per expert a matrix multiply runs at a fraction of a percent of peak compute while saturating memory bandwidth. NVIDIA's own kernel team found that at decode batch sizes it is faster to compute all 256 experts densely and throw away the unselected results than to run a grouped multiply — doing thirty-two times the arithmetic wins. No source carried here.
3. Spreading experts across accelerators costs a round trip per layer. The fix for the first two problems is expert parallelism, which pays for itself in dispatch and combine traffic on every layer of every step. DeepSeek's own analysis puts a ceiling near 67 tokens per second per user on that alone; their production system reports 20 to 22. DeepSeek at ISCA 2025, KV bytes and the all-to-all ceiling ↗
4. Routing is not uniform, and the slowest rank sets the pace. Measured load skew runs four to sevenfold across ranks. Rebalancing experts is worth about 2.5× on decode throughput, which is another way of saying imbalance was eating most of it. No source carried here.

The clean measurement: on identical hardware, engine, precision and request shape, DeepSeek R1 — 37B active of 671B — delivers about 0.94× the per-GPU throughput of a dense 70B model. Sparsity delivered nothing in that configuration. A controlled academic study bracketing an MoE between dense models at both its active and total size found the same shape: matching the small dense model at batch 1, approaching the large one by batch 64. Epoch AI, MoE versus dense at inference ↗Lynx, the batch-size measurement ↗

The arithmetic saving is real — it is the bytes that are not. Everything above is about weight traffic, and the two are different resources. Each token still passes through only its own k experts, so the FLOPs per token stay sparse at any batch size; what stops being sparse is the number of distinct experts the step has to read, because a batch reads the union of what its tokens chose. Sparsity keeps its compute advantage and loses its bandwidth advantage.

Whether that saving is worth anything depends on which resource you have run out of. Decoding is memory-bound, so cheaper arithmetic buys nothing — the step is waiting on bytes, and published estimates put the crossover into compute-bound decoding at batches in the low thousands of tokens, well past what anyone runs. Prefill is compute-bound, and there the saving is immediate and proportional. That is why the workload shape matters more than the architecture: a summariser that reads little and writes at length spends its time in the regime where sparsity does nothing, while an agent replaying a 260,000-token transcript for a 1,100-token reply spends almost all of it in the regime where sparsity pays.

None of this makes sparsity useless. It buys training efficiency, and it buys quality per active parameter. What it does not buy is a proportional cut in serving cost — and the memory bill is set by total parameters regardless, because every expert has to be resident whether or not this step reads it.