Mixture of Experts and Sparse Models – MoE Architecture, Routing and Scaling Laws

The scaling laws for language models (Hoffmann et al., Chinchilla, 2022) established that compute-optimal training requires proportionally scaling both model size and training data. But scaling dense Transformer models — increasing the number of parameters while keeping the compute per forward pass proportional — is expensive: doubling model parameters roughly doubles both training and inference compute. Mixture of Experts offers a way to scale model capacity — the total number of parameters — without proportionally scaling compute: by activating only a fraction of experts per token, MoE models achieve the representational capacity of a large dense model at the inference cost of a smaller dense model. This architectural insight, combined with the engineering advances in expert parallelism and load balancing, has made MoE the dominant architecture for frontier-scale language models. This guide covers MoE architecture and routing, load balancing, expert specialisation, sparse attention as the attention-layer analogue, and the engineering considerations for training and serving MoE models.

MoE Architecture — Routing, Gating and Expert Networks

The standard Transformer block consists of a multi-head self-attention layer followed by a feed-forward network (FFN). In a Transformer MoE block, the single FFN is replaced by K expert FFNs and a gating network G: for each token representation h, the gating network computes softmax logits over K experts, selects the top-k experts by logit, and routes h to those experts: output = Σ_{i ∈ top-k} G_i(h) × E_i(h), where G_i is the gating weight for expert i and E_i(h) is expert i’s transformation of h. In top-1 routing (Switch Transformer), only one expert processes each token; in top-2 routing (Mixtral), two experts process each token and their outputs are combined. Top-2 routing achieves better performance than top-1 at the cost of doubling per-token FLOPs, but since K is typically 8–64, even top-2 activates a small fraction of total parameters.

The gating network is typically a simple linear layer: G(h) = Softmax(W_g h + noise), where the noise term (usually Gaussian noise added during training) encourages exploration and prevents all tokens from routing to the same expert. The linear transformation W_g maps the token representation to K logit scores, one per expert. The softmax over the top-k logits normalises the gating weights to sum to 1 within the active expert set. The gradient flow through the gating network is a key architectural consideration: because routing is a discrete selection, gradients do not flow back through the routing decision itself (only through the gating weights). This makes the gating network’s training a form of combinatorial optimisation that the auxiliary loss regularises, connecting to the reinforcement learning perspective where routing is a policy and expert quality is the reward.

Load Balancing and Expert Specialisation

Without explicit regularisation, MoE training collapses: the gating network learns to route most tokens to the same 1–2 experts (those that first produce slightly lower loss), leaving the remaining experts undertrained. This routing collapse is the MoE analogue of the mode collapse problem in GANs — a rich-gets-richer dynamic where small initial differences in expert quality compound into complete routing imbalance. The auxiliary load-balancing loss prevents this: L_aux = α × K × Σ_{i=1}^K f_i × p_i, where f_i is the fraction of tokens routed to expert i and p_i is the average gating probability assigned to expert i. Minimising L_aux encourages the routing distribution to be uniform across experts. The coefficient α (typically 0.01) balances the load-balancing objective against the primary language modelling loss. Switch Transformer introduced this formulation; subsequent works (GShard, Mixtral) refined it with capacity factor constraints that prevent expert overload during training.

Expert specialisation is an emergent property of MoE training that has been extensively studied. Analysis of routing distributions in trained MoE models reveals consistent patterns: syntactic tokens (punctuation, common function words) are often routed to a small set of high-frequency experts; domain-specific vocabulary (medical terms, legal language, code syntax) is routed to domain-specific experts; rare and complex tokens are routed to higher-capacity “generalist” experts. This specialisation improves model efficiency — each expert develops expertise in a narrow domain, and the gating network learns to route tokens to the expert best suited to process them. The specialisation can be measured by computing the mutual information between token type (or topic distribution) and expert assignment, providing an interpretability signal analogous to the SHAP feature importance for dense models. The connection to clustering algorithms is instructive: the experts in a trained MoE partition the token space into implicit clusters, and the routing function is a soft cluster assignment that can be analysed with the same tools used for mixture model analysis in our Bayesian statistics guide.

Sparse Attention — Longformer, BigBird and Linear Attention

Standard Transformer self-attention is O(N²) in sequence length N, because every token attends to every other token. For a 32,768-token context window (common in 2026 frontier models), this requires over 1 billion attention operations per layer — prohibitively expensive for long documents. Sparse attention models reduce this to O(N) or O(N log N) by restricting which tokens can attend to which. Longformer (Beltagy et al., 2020) combines local window attention (each token attends to w/2 tokens on each side, O(Nw) total) with global attention (a small number of designated global tokens attend to all other tokens, O(Ng × N) total). The local attention captures nearby syntactic and semantic context; the global attention (applied to [CLS] tokens and task-specific tokens) aggregates global document-level information. BigBird adds random attention (each token attends to r random tokens) to the Longformer pattern, improving theoretical expressiveness while keeping the O(N) complexity. The connection to the full Transformer attention mechanism is direct: sparse attention preserves the self-attention interface and can use the same pre-training and fine-tuning infrastructure, just with a different attention mask pattern.

For data scientists, the practical relevance of MoE and sparse models is primarily in model selection and inference cost: when choosing a foundation model for fine-tuning or RAG applications, MoE models (Mixtral, DeepSeek-MoE) offer a better performance-per-inference-FLOP tradeoff than equivalent-compute dense models, while sparse attention models (Longformer, LongT5) enable processing long documents (contracts, scientific papers, code repositories) that exceed the context windows of standard models. The LoRA fine-tuning workflow applies directly to MoE models — LoRA adapters are added to each expert’s linear layers — and the model compression techniques (quantisation, pruning) are applied to each expert independently, with expert-level pruning (dropping entire low-utilisation experts) as an additional MoE-specific compression strategy.

✦ SUMMARIZE THIS ARTICLE WITH AI

ChatGPT Perplexity Claude Grok

Transformer architecture that MoE layers replace the FFN in is in our Transformers guide. LoRA fine-tuning of MoE models — adapters on each expert’s linear layers — is in our LLM Fine-Tuning guide. Model compression — pruning entire low-utilisation experts, quantising expert weights — is in our Model Compression guide. Scaling laws and compute-optimal training that motivate MoE are introduced in our Generative AI guide. Bayesian mixture models as the statistical predecessor of MoE are in our Bayesian Statistics guide. Expert parallelism and distributed training infrastructure are in our Big Data Technologies guide.