Mixture-of-Experts (MoE) routing directs tokens to specific subsets of expert networks. Models like Mixtral 8x7B utilize 13B active parameters out of 47B total. Systems such as gpt-oss-120b select 4 out of 128 experts, while Qwen3-235B-A22B chooses 8 out of 128.
Architectures like DeepSeek-V3 employ shared and routed experts, using 1 shared and 256 routed experts with 8 active. DeepSeek-V3 implements sigmoid scoring with bias balancing to manage expert load. DeepSeek-V4 utilizes sqrtsoftplus scoring and hash routing in its initial layers.
Google’s expert-choice routing allows experts to select top tokens, which can accelerate training convergence by over 2x. However, this approach introduces causal leakage, where expert selection depends on future tokens. Researchers observed an abnormal loss drop of approximately 10% due to this effect.
Dynamic top-k routing enables a variable number of active experts per token. Implementation within SGLang improved P99 time-to-first-token by 1.21 to 1.26x. Developers typically configure models via config.json, adjusting parameters like expert count and top-k methods.
