Modern Large Language Models (LLMs) address performance bottlenecks through a combination of KV cache compression, context length management, and hardware-level kernel optimization. Key techniques include Multi-Query Attention (MQA), Grouped Query Attention (GQA), and Multi-Head Latent Attention (MLA).
DeepSeek V2 introduced MLA, which utilizes compact latent representations instead of full key-value pairs. This architecture reportedly reduced KV cache size by 93.3% and increased generation throughput by up to 5.76 times compared to the company's previous models.
To manage long context costs, models employ sparse attention mechanisms. MiniMax Sparse Attention (MSA) uses an indexer to select top-16 blocks of 128 tokens each, allowing the main attention branch to process only 2048 KV tokens regardless of total context length.
In experiments with a 109B MoE model and a 1M token context on H800 hardware, MSA achieved 28.4 times fewer attention computations, 14.2 times faster prefill, and 7.6 times faster decoding. However, these gains are specific to the research configuration.
While GQA and MQA reduce the memory footprint of the KV cache, sparse methods like MSA focus on reducing the number of tokens processed during attention. Future optimizations are increasingly shifting toward accelerating indexers and block selection processes.
