How a Mixture of Experts Manages Its Experts

Why an MoE depends less on expert specialization than on its ability to balance their use.

Where the problem comes from

A Transformer block alternates between two operations, and only two.

Attention puts tokens in relation with one another. Each token looks at the others, weighs what it finds, and updates its representation based on the context. It is the operation that moves information horizontally, across positions.

The FFN, the feed-forward network, processes each token in isolation. It takes a token representation, projects it into a much larger space, applies a nonlinearity, and projects it back. There is no communication between positions. It is the operation that works vertically, in depth.

Attention, FFN, attention, FFN. Sixty times, eighty times. This is the repeated skeleton of the Transformer.

These two operations do not cost the same. In a Transformer of common size, the FFN contains two thirds of the parameters. This is where the model stores most of what it knows. And because it is applied to every token, every token pays for the whole mass.

This is the point of tension that the Mixture of Experts attacks. It does not touch attention. It replaces the FFN.

What exactly is an expert?

An expert is not a specialized module that someone designed, trained, and inserted. It is simply one FFN among N, with exactly the same structure as its neighbors, initialized randomly like them, and trained at the same time as them.

A mini-model, then. But a mini-model whose contents we do not know.

Its weights are matrices containing millions of coefficients with no readable structure. There is no label and no way to open expert 412 and discover that it « handles Python code ». We can observe when it activates. Patterns often emerge that are lexical or syntactic. But an activation correlation is not a functional description. And on the other side, what it receives is the result of a decision made by a router that was also learned, according to criteria nobody specified.

The word « expert » is therefore misleading, and we should be wary of it throughout what follows. It describes a position in an architecture, not an identified skill.

The bet, and its condition

The MoE bet can be summed up in one sentence: decouple capacity from cost. Build a huge network, but activate only a fraction of it for each token. A 2.8-trillion-parameter model that computes only 50 billion per token.

Put that way, it sounds like a free lunch. It is not, and we should state immediately where the bill appears.

The bet only works if tokens are distributed roughly evenly among experts.

This is the central condition and the thread running through this article. An MoE in which 5% of experts absorb 80% of the traffic is not a large efficient model. It is a small model dragging a huge amount of dead parameters behind it. The capacity we paid for is unused. And, as we will see, the throughput of the entire cluster collapses on top of that.

Almost everything that follows, from routing mechanisms to successive fixes and infrastructure choices, comes from this single requirement: obtain a balanced distribution, and obtain it early, before the initial imbalance has hardened.

Anatomy of a routing decision

In an MoE block, the single FFN is replaced by N experts plus a router, typically a simple linear layer.

For a hidden token representation h:

1. The router computes one score per expert: s = softmax(h · W_r), a vector of dimension N
2. The top-k scores are selected
3. The token is sent to the k corresponding experts
4. Their outputs are recombined, weighted by normalized scores

The mechanism fits in four lines. Almost all of the difficulty lies elsewhere.

Three properties of this decision determine everything that follows.

It is hard. In practice, only the selected experts work on the token and receive a learning signal. An expert that is rarely selected therefore improves less, which can make it even less attractive.

Mechanically, top-k is a partial argmax, discrete and non-differentiable. We do not backpropagate through « which expert was chosen ». The gradient only travels through the recombination weights, meaning the scores of the experts actually selected. There is nothing to propagate for the others, and that is precisely the mechanism by which a neglected expert remains neglected.

It is local. Each block has its own router, trained independently. A token passing through sixty blocks makes sixty decisions without coordination between them. There is no coherent path through the network and no global plan.

It is per token, not per request. This is the fundamental difference from multi-model routing, such as RouteLLM, Mixture-of-Agents architectures, or any dispatcher that sends an entire request to a model. There, a bad decision ruins the whole answer. Here, each token in each layer makes its own choice, and one mistake is diluted among tens of thousands of others. The two systems share the word « routing » and little else. Do not transfer results from one to the other.

Router collapse

The router is trained from scratch, without supervision about what a « good » assignment would be. The only signal is the model’s global loss.

That creates a vicious feedback loop.

At the very beginning of training, by pure initialization chance, one expert receives slightly more tokens than its neighbors. It therefore trains a little more. It becomes a little better. The router, optimizing the loss, assigns it slightly higher scores. It receives even more tokens.

This phenomenon is called router collapse, and it reinforces itself. It happens during the first few thousand steps, and once established it does not reverse: an untrained expert has no reason to become attractive.

The root cause is structural. Nothing in the training objective rewards diversity of use. The loss only measures prediction quality. A model that uses only 5% of its experts but predicts correctly is, from the loss’s point of view, a successful model.

That is why all the solutions in the next section are constraints added from outside. None of them emerges from the optimization itself.

What imbalance costs, and how to measure it

There are two distinct costs, and they require different measurements.

Capacity cost

If a fraction of the experts does all the work, you paid for a huge model and are using the effective capacity of a much smaller one.

How to measure it. Instrument the router on a representative sample of real traffic, count activations per expert, and calculate the entropy of the resulting distribution. The theoretical maximum is log₂(N), reached when all experts are used equally. Entropy far below that signals a model that does not use what you paid for. Do this layer by layer: collapse is not uniform through depth.

A frequently forgotten sizing corollary: inactive parameters still occupy memory. A model with « 50 billion active » parameters is not deployed like a dense 50-billion-parameter model. The whole set of experts, active or not, must be hosted.

Throughput cost

This one is brutal, and surprising.

Experts are spread across dozens or hundreds of accelerators. This is expert parallelism. Each step requires an all-to-all communication: tokens travel to the GPUs hosting their experts, are processed, and then come back.

If one GPU receives ten times more tokens than average, everyone else waits for it. The throughput of the entire cluster is dictated by the most heavily loaded node. A 2x imbalance does not cost 2% performance. It can cost half the throughput.

How to measure it. The advertised activation ratio gives you FLOPs, not latency. Measure actual throughput on your topology and compare it with the theoretical throughput derived from the number of active parameters. The gap is communication.

This is the point architects most often underestimate. Load balancing in an MoE is not primarily a model quality issue. It is an electricity bill issue.

The solutions, and what they cost

For what follows, here is an image. Imagine a sorting workshop: packages arrive continuously, N workstations process them, and a foreman decides where to send each package. The foreman learns on the job, without instructions, by observing only whether overall production improves.

Left alone, it sends everything to its three favorite stations. The others watch.

The fine (2017)

The principle. At the end of each day, we measure the deviation from an equal distribution and fine the foreman proportionally. This is the auxiliary loss from the foundational paper by Shazeer and colleagues: a regularization term added to the loss that penalizes imbalance.

It works. The foreman quickly learns to spread the load.

The limit. It now has two bosses giving contradictory orders. One wants production, the other fairness. Yet an uneven distribution can be perfectly justified. The token distribution in real text is not uniform. By forcing uniformity, we sometimes prevent the foreman from making the right choice.

And the size of the fine becomes a troublesome hyperparameter: too small and collapse happens, too large and quality degrades. Its optimal value drifts during training.

The fixed-size bin (2020-2021)

The principle. We stop thinking in terms of incentives and install a physical constraint: each workstation gets a bin with a fixed capacity. When the bin is full, later packages are not processed and go straight to the exit. This is the capacity factor of GShard and Switch Transformer, with token dropping: excess tokens skip the MoE block and pass through the residual connection.

What it really fixes. The throughput problem, decisively. The bins have a known size, so buffers have static dimensions, kernels can be compiled once, and communications can be planned. That is the mechanism’s real purpose, much more than imbalance control itself.

The limit. Some packages are not processed. And bin size becomes another parameter to tune, with a direct tradeoff between wasted memory when the bin is too large and dropped tokens when it is too small.

The thumb on the scale (2024)

The principle. We no longer punish the foreman or limit the bins. We quietly alter its evaluation: each workstation gets a small bonus or penalty added to its score at selection time. An overloaded station gets a penalty, an ignored station gets a bonus. This is the loss-free balancing introduced by DeepSeek-V3: a per-expert bias added to routing scores before top-k selection and adjusted by a heuristic rule outside backpropagation.

The advantage is decisive. The loss is no longer polluted. The foreman no longer has two bosses. It keeps optimizing production alone, while balancing happens beside it through the scale adjustment. This is what made the approach dominant.

The limit. We have not removed tuning, only moved it. We still need to decide how quickly to adjust the thumb. Too slowly and imbalance settles before it is corrected. Too quickly and routing oscillates: a station becomes attractive, overloads, is penalized, empties, and becomes attractive again. Training becomes unstable.

Each generation therefore solves the previous generation’s problem by moving the hyperparameter rather than removing it. This is exactly where the case study will pick things up.

Many small ones instead of a few large ones

One sizing question remains: how many experts?

Practice gives a striking answer. The first popular MoEs had eight experts and activated two. Recent models have several hundred and activate around fifteen. The activation ratio has fallen from 25% to less than 2%.

Why this race? Not to have more specialties. To have more recipes.

Let us return to the kitchen. An MoE with eight large experts is a menu of eight ready-made dishes from which we choose two. The number of possible menus is quickly exhausted, and if none of the eight dishes exactly fits what we want, so be it.

An MoE with nine hundred small experts is a pantry of nine hundred ingredients from which we combine sixteen. No individual ingredient makes a dish, but the number of possible recipes becomes astronomical. At a constant parameter budget, we have not added capacity. We have added finer assembly.

This is the important conceptual point, and it casts what we said about the nature of an expert in a new light. Expressiveness comes from combinatorics, not individual specialization. An expert does not need to embody anything identifiable. The mixtures only need to be discriminative.

The number of active experts also increases, from two to sixteen. This makes sense: with smaller ingredients, more are needed to compose a complete dish. We trade size for number.

And the price is the one from the previous section. The larger the pantry, the more likely it is that we always pick from the same aisle. Every jump in granularity must come with progress in balancing, or it will not hold. The thread is still there.

Shared experts

A refinement introduced by DeepSeekMoE and widely adopted since then is to mix two categories in the same block.

A shared expert processes every token without going through the router. It is always active. Routed experts are selected by top-k as described above.

Why? In a pure MoE, every expert has to relearn generic skills: basic syntax, surface regularities, and everything needed regardless of content. This redundancy is a major waste. The same knowledge is duplicated hundreds of times.

The shared expert factorizes this common foundation. Routed experts are then free to encode only the difference. Returning to the workshop, one station handles every package’s basic operations, while the others intervene only when something more is needed.

It is also a safety net. Even if routing degrades, the shared expert continues processing every token. Quality drops, but the model does not collapse.

What this means for infrastructure

Three constraints follow from the mechanism, and no modeling trick avoids them.

All-to-all dominates. At every block, tokens travel to the GPUs hosting their experts and come back. This all-to-all communication consumes bandwidth and is repeated at every layer. That is why fast interconnect domains matter: it is better for the exchange to remain inside a tightly coupled group of accelerators.

Static shapes are worth their weight in gold. If buffer sizes depend on the observed distribution, memory must be reallocated and sometimes kernels recompiled at every step. Fixing sizes at compile time makes it possible to optimize kernels once and for all.

The CPU must stay out of the critical path. If the GPU waits for a host decision at every MoE block, the latency of that loop dominates everything else. Serious implementations keep dispatch entirely on the accelerator.

Case study: Kimi K3

Kimi K3, released by Moonshot AI on July 16, 2026, is the first open 3T-class model. It pushes everything above to an unprecedented scale and offers an original answer to the balancing problem.

896 experts, with 16 active per token. Around 50 billion active parameters out of 2,800, for an activation ratio below 2%. The block is called Stable LatentMoE, and the published architecture diagram shows the two categories described above: shared experts and routed experts. This doubles the granularity compared with Kimi K2 and quadruples it compared with DeepSeek-V3.

At this level of sparsity, Moonshot explicitly says that routing and optimization become first-order concerns. Its answer is called Quantile Balancing: expert allocation is derived directly from router-score quantiles, eliminating heuristic updates and a sensitive balancing hyperparameter.

Let us return to the workshop one last time. The thumb on the scale reasoned in absolute value: « this station is too busy, subtract 0.05 from its score ». That creates the unsolvable question of how much to subtract and how quickly, a quantity whose correct value depends on the scale of the scores, which itself drifts during training.

Reasoning in quantiles means stopping the practice of scoring stations and instead ranking them. We no longer ask « what is this station’s score? » but « where does it stand in today’s ranking? ». Allocation follows from that position.

The advantage is scale invariance. A rank does not change if every score is multiplied by two. All the sensitivity that made tuning difficult disappears because the quantity being manipulated is normalized by construction. There is no correct value to find: the empirical distribution provides its own reference at every step.

The lesson goes far beyond MoE: when a hyperparameter is structurally painful to tune, it is often better to change the quantity being manipulated than to perfect the tuning. Moving from value to rank removes the problem instead of optimizing it.

The technical report has not yet been published. It will accompany the weights announced for July 27, 2026. The above relies on the launch blog, which gives the principle rather than the algorithm. The interpretation in terms of ranking is mine. The release of the weights will also make it possible to directly measure usage across the 896 experts, which is the most honest test of the claim.

Opening: 896 black boxes

This article has spent its time circling a question that must finally be named.

We know how to build an MoE. We know how to diagnose its collapse, balance it, and make it run on a cluster. What we do not know is what is inside it.

An expert remains what we placed at the beginning: a mini-model whose weights are not readable and whose behavior cannot be characterized. Multiplying it by 896 changes nothing, except that we now have 896 black boxes instead of one, plus a router that chooses between them according to criteria nobody specified.

This is not a gap in curiosity. It is a gap in tooling. We measure activation distributions, how often each expert is called, but a distribution is not an interpretation. Knowing that expert 412 receives 0.11% of traffic tells us nothing about what it does.

This is the territory of approaches that seek to characterize a model’s internal space rather than only its outputs: identifying meaningful directions in latent space, finding structures that persist from one layer to the next, and understanding what is shared between modules and what remains local. Work on shared representation spaces and global workspace architectures asks precisely this question: how does information become available to an entire system rather than remaining confined to one module?

An MoE is a natural application ground. It is an explicitly modular system, with a clear boundary between modules, an observable selection mechanism, and a residual stream that acts as a shared channel. Almost a test bench.

Kimi K3 makes this concrete for the first time at this scale: 896 experts, open weights, and a community that will finally be able to look inside.

That will be another question. Not how to balance experts, but how to understand what they actually learn.

Summary

• An MoE replaces the FFN in a Transformer block, not attention. That is where two thirds of the parameters are.
• An expert is an opaque mini-model. We can observe when it activates, but not describe what it does.
• The bet, decoupling capacity from cost, only works if tokens are distributed evenly. Everything else follows from that.
• The routing decision is hard, local, and per token. It has little to do with inter-model routing despite the shared vocabulary.
• Router collapse is self-reinforcing and happens early. Nothing in the loss prevents it. All solutions are external constraints.
• Imbalance costs both capacity and throughput. In expert parallelism, the most heavily loaded node sets the pace for the entire cluster.
• Expressiveness comes from combinatorics. Many small ingredients are more useful than a few large dishes.
• Kimi K3’s contribution is Quantile Balancing: rank instead of score, removing a hyperparameter instead of tuning it.

Resources

The origins

• Jacobs, Jordan, Nowlan & Hinton (1991), Adaptive Mixtures of Local Experts, Neural Computation 3(1): the foundational idea, thirty years before Transformers.
• Jordan & Jacobs (1994), Hierarchical Mixtures of Experts and the EM Algorithm, Neural Computation 6(2): the hierarchical version.
• Vaswani et al. (2017), Attention Is All You Need: arXiv
  .03762. The attention / FFN alternation discussed in the introduction.

Sparse MoE and balancing

• Shazeer et al. (2017), Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer: arXiv
  .06538. The foundational paper and the auxiliary loss.
• Lepikhin et al. (2020), GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding: arXiv
  .16668. Capacity factor and automatic sharding.
• Fedus, Zoph & Shazeer (2021), Switch Transformers: arXiv
  .03961. Top-1 routing pushed to its limit, with token dropping.
• Zoph et al. (2022), ST-MoE: Designing Stable and Transferable Sparse Expert Models: arXiv
  .08906. On training instability, still the best entry point.
• Wang, Gao, Zhao, Sun & Dai (2024), Auxiliary-Loss-Free Load Balancing Strategy for Mixture-of-Experts: arXiv
  .15664. The thumb on the scale: per-expert bias without gradient interference.

Granularity and shared experts

• Dai et al. (2024), DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models: arXiv
  .06066. Fine segmentation and shared expert isolation, the two ideas since adopted by almost everyone.
• Jiang et al. (2024), Mixtral of Experts: arXiv
  .04088. The eight-large-expert MoE, with a useful routing analysis.
• DeepSeek-AI (2024), DeepSeek-V3 Technical Report: arXiv
  .19437. 671 billion parameters, 37 active, and production auxiliary-loss-free balancing.

Overviews

• Fedus, Dean & Zoph (2022), A Review of Sparse Expert Models in Deep Learning: arXiv
  .01667.
• Cai et al. (2024), A Survey on Mixture of Experts: arXiv
  .06204.

Toward openness

• Probing Semantic Routing in Large Mixture-of-Expert Models: arXiv
  .10928. The question stated directly: does routing encode semantics, or only surface patterns?

Kimi K3

• Kimi K3 launch blog (July 16, 2026): source for all figures and mechanism names cited here. The technical report and weights are announced for July 27, 2026.

Read More

• The End of Transformers? Hybrid Attention and State-Space Models in 2025
• DeepSeek V4: The Lab Drawing the Blueprints for Future LLMs
• Google Opens the Hood of LLMs