The Mixture-of-Experts (MoE) architecture allows large language models to decouple the total number of parameters from the computational cost per token. While dense models require full computation for every parameter, MoE models activate only a subset of experts per token, significantly increasing processing speed.
In MoE models, the feed-forward network is divided into multiple smaller experts. A router selects the top-k experts for each token, while the remaining experts remain inactive. This approach allows models like Qwen 3.5 to utilize 397 billion parameters while performing computations equivalent to 17 billion parameters.
Research indicates that MoE experts do not specialize in specific topics like medicine or law, but rather focus on token patterns such as syntax, punctuation, or code structure. To prevent the 'dead expert' problem, where some experts are never selected, developers use auxiliary loss functions to balance the workload across all available experts.
Despite the computational efficiency, MoE models require all parameters to be loaded into VRAM because any expert might be needed at any time. For instance, Qwen 3.5 requires approximately 794 GB of memory in BF16 format. Furthermore, the performance advantage diminishes with large batch sizes as the memory bandwidth requirements increase.
