Mixture of Experts

Mixture of Experts (MoE) architectures, particularly sparse MoEs, have gained prominence for scaling large models efficiently. In a sparse MoE, the gating network selects a small, fixed number (often top-k) of experts for each token or input. This contrasts with 'dense' MoEs where all experts contribute to the final output via a weighted sum. The gating network typically outputs probabilities or scores over the experts, which are then used to select and weight the active experts. For instance, in a Transformer-based MoE, the feed-forward network layer is replaced by multiple MoE layers. Each MoE layer contains multiple feed-forward 'experts,' and a gating function routes each token to a small subset (e.g., 2) of these experts. This sparsity allows for a massive increase in the total number of parameters (model capacity) without a proportional increase in computational cost per token during inference. Key challenges include load balancing (ensuring all experts receive roughly equal amounts of training data) and auxiliary loss functions (e.g., load balancing loss) are often employed to encourage uniform expert utilization. Expert collapse, where the gating network consistently favors only a few experts, is a common failure mode. Theoretical analysis often focuses on the properties of the gating function and the optimization dynamics of such sparse, high-dimensional systems.