Top-k gatingAuxiliary load-balancing lossExpert parallelismCapacity factor
Mixture-of-Experts: Decoupling Parameters from Compute
1 min readLearnJul 17, 2026
Key Takeaway for Students
Sparse MoE layers route each token to a small subset of expert FFNs via a learned gate, so total parameters grow while per-token FLOPs stay roughly constant. Routing instability, expert collapse and load balancing remain open problems.
Reading in Plain English mode — toggle below to switch
Sparse MoE layers route each token to a small subset of expert FFNs via a learned gate, so total parameters grow while per-token FLOPs stay roughly constant. Routing instability, expert collapse and load balancing remain open problems.