Mixture-of-Experts (MoE) Models and Expert Routing

Updated on Oct 15, 2026

9 min read

Large language models can contain far more parameters than they use to process any one token. Mixture-of-Experts (MoE) architecture makes this possible by routing each token through a small, selected group of neural-network experts instead of activating every feed-forward network. This explainer is for developers and infrastructure engineers who need to understand MoE routing, its compute and memory trade-offs, and how expert parallelism affects model serving.

What Is a Mixture-of-Experts Model?

A mixture-of-experts model contains multiple learned sub-networks called experts and a router that chooses which experts process each input. In a Transformer, the experts commonly replace the ordinary feed-forward network (FFN) in some or all layers. The attention layers and other shared components may remain dense: they still run for every token.

The router computes a score for a token and selects one or more experts. Each selected expert transforms the token representation; the model combines the resulting vectors, often weighting them by the router’s scores, before continuing through the next layer. The Switch Transformer research paper describes a sparse design that routes each token to one expert. Other models route each token to multiple experts. In either case, the model’s parameters are conditional: only part of the expert network is active for a token.

Why MoE Models Exist

In a dense Transformer, each token passes through the same FFN in every layer. Increasing the model’s capacity therefore increases the work performed on every token. MoE separates these two quantities: a model can store many expert parameters while activating only a fraction of them for a particular token.

That separation can improve the amount of learned capacity available for a given per-token computation budget. It does not make a large model inexpensive in every sense. The expert weights still have to be stored in accelerator memory or accessed from elsewhere, and the router and serving system must send tokens to the right experts. A model with many total parameters may need more hardware memory and communication even if it activates relatively few parameters per token.

MoE is useful when the added model capacity benefits a workload and the infrastructure can support sparse execution. It is not an automatic shortcut to lower latency, cheaper inference, or better answers. Those outcomes depend on the model, batch, routing balance, hardware, and serving implementation.

How MoE Routing Works

For each token representation, a learned router produces a score for each available expert. A top-k selection step identifies the experts that will process the token. With top-1 routing, one expert is selected; with top-2 routing, two experts are selected and their outputs are typically combined using normalized routing weights.

token representation
        |
        v
    router scores
        |
  select top-k experts
    /             \
 expert A       expert B
    \             /
 weighted combination
        |
        v
 next Transformer layer

Routing is generally performed per token, not once for an entire prompt or request. As a result, tokens from one input can go to different experts, and the same expert can receive tokens from many requests in a batch. A model may also use shared experts that process tokens alongside routed experts. The exact routing rule, number of experts, and expert placement are model-specific.

Property	Dense feed-forward network	Mixture-of-Experts feed-forward network
Sub-network used per token	The same FFN processes every token	A router selects one or more experts
Parameter activation	Most FFN parameters are used on each token	Only selected expert parameters are active for each token
Total parameter storage	Grows with the dense model’s width and depth	Can be large because all experts’ weights must be represented
Per-token compute	Generally grows with FFN size	Depends on active experts, routing, and shared layers
Communication needs	Often contained within the layer’s parallel group	May require moving token data to remote experts and returning outputs
Main operational challenge	Compute and memory scaling	Memory capacity, load balance, and dispatch communication

Total parameters and active parameters per token describe different things. Active-parameter counts can help compare compute, but they do not say how much memory is required to hold the model. Likewise, sparse activation does not mean that only a small set of experts needs to be loaded: any expert could be selected by a later token.

Components and Key Concepts

Experts are learned functions, commonly FFNs with their own weights. They are not hand-written specialists or independent models with guaranteed labels such as “math expert.” Training may lead experts to respond differently to input patterns, but interpreting an expert’s role from its index alone is unreliable.

The router or gate scores candidate experts from each token’s hidden state. Its decisions are part of the model’s forward pass. A router can become imbalanced, sending too many tokens to a small number of experts while leaving others underused. Training approaches such as auxiliary load-balancing losses, and inference techniques such as routing biases, can encourage more even use. The particular mechanism depends on the architecture and training recipe.

Capacity and token overflow describe limits on how many tokens an expert can process in a batch. Some implementations set a capacity factor and may drop or reroute excess assignments when an expert is overloaded. Other systems aim for dropless execution and handle variable loads differently. Avoid assuming all MoE models share the same overflow behavior; it can affect both model quality and throughput.

Expert parallelism (EP) distributes different experts across devices. When a token is assigned to an expert on another device, the serving system dispatches its representation there and gathers the expert output back. This often involves collective communication such as all-to-all exchanges. EP is distinct from tensor parallelism, which divides the computation or parameters of an individual layer across devices, and data parallelism, which runs replicas on separate data. Systems can combine these strategies.

The vLLM expert-parallel deployment guide documents how its serving engine places experts and enables EP. The important systems trade-off is that more expert capacity can require more cross-device traffic. Network bandwidth, GPU topology, batch composition, and routing balance all influence whether expert placement helps or hinders serving.

Real-World Uses and Trade-Offs

MoE models are used in large language-model training and inference when developers want a high-capacity model with conditional computation. A model can route different token representations through different parts of its FFN, while shared attention and other layers maintain common processing. The vLLM project provides an inference engine with expert-parallel deployment support for MoE models.

At serving time, expert parallelism can distribute a model that would not fit comfortably on one accelerator. But a token’s path may cross devices for multiple MoE layers, so expert dispatch can add latency and consume interconnect bandwidth. A larger batch may improve utilization and spread communication costs across more tokens, but can increase queueing time and memory use. A single-GPU or single-node result may not predict the behavior of a multi-node deployment.

Operational evaluation should include:

Quality: compare outputs on representative prompts because routing changes are not merely hardware settings.
Latency: measure time to first token and inter-token latency, not only model throughput.
Throughput and balance: track tokens per second alongside per-expert token counts and utilization.
Memory: account for total expert weights, temporary activations, and per-request attention cache separately.
Communication: inspect collective time and interconnect utilization when experts are split across devices.

MoE also affects capacity planning. Weight quantization can reduce the storage required for model parameters, but it does not change which experts are selected or remove dispatch traffic. For the distinction between weight memory and attention state, see LLM inference optimization and KV caching. LLM quantization for local inference covers weight formats and their trade-offs.

Getting Started: Serve an MoE Model with Expert Parallelism

The following example uses vLLM to serve a supported MoE model on a Linux system with four compatible GPUs. Model repositories can require access approval, and the model must fit the available memory; check the model card and vLLM’s GPU installation guide before installing. On a supported environment with uv installed:

uv pip install vllm --torch-backend=auto

vllm serve Qwen/Qwen3-30B-A3B \
  --tensor-parallel-size 4 \
  --enable-expert-parallel

Here, tensor parallelism is configured across four GPUs and vLLM enables expert parallelism for the model’s MoE layers. The exact placement and supported parallel sizes depend on the model, vLLM version, accelerator topology, and cluster configuration. Consult the current deployment guide rather than treating the example as a universal production configuration.

Check that the server is ready and send a small request:

curl http://localhost:8000/v1/models

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"Qwen/Qwen3-30B-A3B","messages":[{"role":"user","content":"Explain expert routing in one sentence."}],"max_tokens":64}'

For a meaningful comparison, run the same model and representative prompt mix with and without expert parallelism where each configuration is supported. Keep GPU count, concurrency, prompt lengths, and output limits controlled. Record time to first token, inter-token latency, tokens per second, GPU memory, expert utilization, and communication time. If one or two experts receive a disproportionate share of tokens, investigate routing balance and batch composition before adding devices. If communication dominates, test a topology that keeps experts closer together or a different expert-parallel size.

The setup command is an operational starting point, not a benchmark result. A successful request confirms that the endpoint loaded and responded; it does not prove that the model meets quality, latency, or reliability requirements for production.

Common Misconceptions

“Only active parameters need memory.” Active parameters describe what computes a token’s output. The full model’s weights still need to be represented and available to the serving system.
“Each expert is a human-readable skill.” Experts are learned components selected by a router. Their apparent specialization is not a dependable label or guarantee.
“MoE always makes inference faster.” Sparse computation can reduce per-token FFN work, but routing, memory access, expert imbalance, and inter-device communication can offset that benefit.
“Expert parallelism is the same as tensor parallelism.” EP distributes experts; tensor parallelism splits layer computations or weights. A serving configuration may use both.

Transformer Architecture Deep Dive explains the attention and feed-forward layers that MoE changes.
LLM Serving Schedulers and Continuous Batching covers the scheduling and latency concerns around inference workloads.
LLM Inference Optimization and KV Caching Explained explains attention-state memory, which is separate from model weights.
LLM Quantization for Local Inference describes reducing model-weight storage and its trade-offs.

Changelog and Last Updated

Initial publication.

Last updated: October 15, 2026