Mixture of experts (MoE) is an efficient architecture, but it is often misunderstood when compared with dense models. Each layer holds many feed-forward blocks, the experts, and a router picks a few of them for each token. For example, Gemma4-26B-A4B has 128 experts per layer and picks 8 per token, so each token uses about 4B parameters out of 26B. However, that figure invites two inaccurate conclusions, namely that the MoE model needs the memory of a dense 4B model and that both cost the same to serve. I measured the memory and cost of this model on an A100, against a dense model.
Misconception 1: “MoE Needs the Memory of a Dense 4B Model”
Every expert has to stay in GPU memory, because any token can pick any of them. So memory follows the full model size rather than the active size. At BF16, 4B parameters would take about 7.5 GiB, while Gemma4-26B-A4B takes 48.5 GiB.
The dense model I compare with is Gemma4-E4B. The E stands for effective, so it has about 4B effective parameters but stores about 8B, which takes 15.2 GiB. That is still about a third of the MoE’s memory. vLLM reports the following when each model starts on one A100.
| Gemma4-E4B (dense) | Gemma4-26B-A4B (MoE) | |
|---|---|---|
| Memory to load the model | 15.2 GiB | 48.5 GiB |
| Memory left for the KV cache | 56.6 GiB | 23.1 GiB |
| vLLM’s maximum concurrency at 32,768 tokens per request | 94.5 | 19.0 |
Long requests therefore reach the KV cache limit far sooner on the MoE. Only part of this gap comes from loading more weights, which leaves 2.5 times less memory for the KV cache. The MoE model also caches about 2.5 times as many bytes per token, because of its attention layout rather than its experts. Both models use a sliding window in most layers, so vLLM can fit more requests than the cache size divided by the request length suggests.
Misconception 2: “MoE Costs Like a Dense 4B Model”
This holds for one request at a time, but not under load. The following chart shows time per output token (TPOT) as concurrency grows, on one A100. With only one request, the two models cost about the same per token. At 32 concurrent requests, the MoE takes about twice as long per token as the dense model.
Larger Batches Read More Experts
The 4B in the name Gemma4-26B-A4B counts the parameters one token uses (the A stands for active). A decode step, however, serves every request in the batch, and each token can pick different experts, so the step reads every expert that at least one of them picked. A dense model reads all its weights on every step, whatever the batch size, while an MoE reads more experts as the batch grows.
I counted the distinct experts inside vLLM, and split the time of a decode step into the expert kernels, which is the GPU time spent in the expert layers, and the rest of the step.
| Batch size | Experts read per layer (of 128) | Expert weights read per step | Expert kernels | Rest of the step | Whole step |
|---|---|---|---|---|---|
| 1 | 8.0 | 2.9 GB | 2.0 ms | 6.6 ms | 8.7 ms |
| 8 | 31.5 | 11.2 GB | 7.1 ms | 7.7 ms | 14.9 ms |
| 32 | 56.2 | 20.0 GB | 12.3 ms | 8.9 ms | 21.5 ms |
From batch 1 to 32, the expert weights read grow 7 times and the time in the expert kernels grows 6 times, while the rest of the step grows only 1.3 times. The experts go from about a quarter of the step to more than half of it, and the whole step takes 2.5 times as long. From the expert counts, the expert kernels read their weights at about 1.4 to 1.6 TB/s. That is close to the memory bandwidth a plain copy measured on this A100, 1.6 TB/s, so their time is set by the bytes read.
The rest of the step grows about as much as the dense model’s TPOT does over the same range, which rises by about a third. That resemblance is only suggestive, since I did not profile the dense model’s step. These fixed batches also differ from real serving, so they explain much of the gap under load rather than all of it.
When serving, before choosing an MoE over a dense model, measure TPOT at the load you plan to run, and check that the KV cache holds your requests.
Setup
In these experiments, I served the models, Gemma4-E4B and Gemma4-26B-A4B, with vLLM server 0.19.0 on one A100 (80 GB), in BF16. The concurrency numbers come from the 201 test questions of TeleMath, without reasoning, with two runs at each concurrency cap and a maximum length of 32,768 tokens. The MoE ran with vLLM’s default kernel configuration, because vLLM ships no tuned one for this model on an A100.
The expert counts and step times use fixed batches with these settings:
- Every request starts together and decodes exactly 128 tokens, and prefill is left out.
- The expert counts run without CUDA graphs, because forward hooks do not fire inside them. This changes the speed but not which experts are picked.
- The step times run with CUDA graphs, as in serving.
Serving uses continuous batching and chunked prefill, so the batch size there varies from step to step. This covers one MoE on one GPU type, so other models and GPUs may differ.