A mixture of experts (MoE) is a model design that contains many specialized sub-networks — the “experts” — but activates only a small subset of them for each token, chosen by a lightweight router. This lets a model have a very large total parameter count for capacity while only doing a fraction of the computation per token, making it far more efficient than a dense model of equivalent size. Several leading open and closed LLMs now use MoE architectures.
Key takeaways
- MoE splits the network into many experts and uses a router to activate only a few per token.
- This decouples total capacity from per-token compute: huge parameter counts, but only a slice runs at a time.
- The result is strong quality at lower inference cost — but higher memory needs, since all experts must be loaded.
- MoE is a key reason recent large open-weight models deliver frontier-level quality on manageable hardware.
Mixture of experts, defined
In a standard “dense” transformer, every parameter participates in processing every token. A mixture of experts changes this. It replaces some layers with a set of parallel expert networks and adds a small gating network, or router, that decides which experts should handle each token. Typically only two of many experts are activated per token.
The payoff is a favorable tradeoff between capacity and cost. A model might hold hundreds of billions of parameters in total — giving it broad knowledge and skill — while only a modest fraction of them actually run for any given token. You get much of the quality of a giant dense model at a small fraction of the per-token compute.
How the router chooses experts
The gating network is the heart of an MoE. For each token, it produces scores over the available experts and selects the top few to activate. Those experts process the token, and their outputs are combined — weighted by the gating scores — into the layer’s result. Over training, experts tend to specialize, with different ones becoming adept at different kinds of tokens or patterns.
A practical challenge is load balancing. If the router funnels most tokens to a handful of popular experts, the rest are wasted and the popular ones become bottlenecks. MoE training therefore includes balancing objectives that encourage the router to spread work evenly. This internal, per-token routing is conceptually distinct from — though it rhymes with — LLM routing between whole models.
The compute-versus-memory tradeoff
MoE’s efficiency comes with a catch: memory. Although only a few experts compute per token, all experts must be resident in memory because any of them might be needed for the next token. So an MoE model uses far less compute than a dense model of the same total size, but roughly the same memory footprint. You trade FLOPs for VRAM.
This shapes deployment decisions. MoE is attractive when you are compute- or latency-bound and can afford the memory, since you get high quality per unit of compute. It is less attractive when GPU memory is the tight constraint. Weighing this tradeoff against your hardware is exactly the kind of decision that determines which model fits a given environment.
Why MoE matters for enterprise AI
Mixture of experts is one of the main reasons recent open-weight models punch above their weight. By separating capacity from active compute, MoE lets organizations run models with frontier-class quality at inference costs that were previously out of reach — an important enabler for on-premise and cost-sensitive deployments.
For an enterprise choosing what to self-host, recognizing whether a model is dense or MoE clarifies its resource profile: expect large memory requirements but efficient throughput. That understanding feeds directly into GPU sizing, capacity planning, and the routing strategy that decides which requests deserve a big MoE model versus a lean dense one.
From concept to a governed, on-premise reality
VDF AI supports open-weight MoE models alongside dense ones, running them inside your own environment. Because MoE trades compute for memory, VDF AI helps size GPU memory correctly so you capture MoE’s efficiency without hitting VRAM ceilings.
At the fleet level, VDF AI’s model router treats a large MoE model as one option among many — reserving it for requests that justify its footprint and sending routine work to smaller dense models, so you get frontier quality only where it earns its keep.
Frequently asked questions
What is a mixture of experts (MoE)?
It is a model architecture made of many specialized sub-networks, or experts, where a small router activates only a few experts per token. This gives the model a very large total capacity while keeping per-token computation low.
How is MoE different from a dense model?
In a dense model, every parameter is used for every token. In an MoE model, only a small subset of parameters (the selected experts) runs for each token, so the model can be far larger in total while doing much less compute per token.
Does MoE make models cheaper to run?
It reduces compute per token, which lowers latency and compute cost. However, all experts must be held in memory, so MoE models require roughly the same memory as a dense model of equal total size. It trades compute savings for higher memory needs.
What does the router in an MoE do?
The router, or gating network, scores the experts for each token and selects the top few to activate. Their outputs are combined into the layer result. Training encourages experts to specialize and balances load so no single expert is overused.
Is MoE routing the same as LLM routing?
No, though they are conceptually similar. MoE routing happens inside a single model, choosing experts per token. LLM routing chooses between entirely separate models for a whole request based on cost, latency, quality, and policy. They operate at different levels.
Put these concepts to work on infrastructure you control.
VDF AI runs governed agents, private retrieval, and model routing inside your own cloud, data center, or air-gapped network. Book a walkthrough mapped to your stack.