The Kimi K3 model is an open-source AI architecture that competes with closed-source industry leaders. Its weights are publicly available for deployment, and it performs competitively on independent benchmarks against models from GPT and Anthropic.
To handle a context window of one million tokens without memory exhaustion, developers replaced the standard KV-cache with a fixed-size state matrix. They implemented a delta-rule mechanism to update this state, using a coefficient alpha constrained between e⁻⁵ and 1 to prevent numerical instability in BF16 precision.
The model utilizes a custom FlashKDA kernel built on NVIDIA's CUTLASS library to parallelize state updates across GPU streaming multiprocessors. For long-context prefilling, an automated scheduler partitions text into segments, allowing parallel computation of state transitions across different GPU blocks, which are later reconstructed using prefix-scan.
Kimi K3 employs a hierarchical layer structure where layers are grouped into blocks of 12. Instead of passing results sequentially, each block uses a learned pseudo-query to selectively retrieve information from previous blocks, reducing memory overhead compared to storing every individual layer's output.
The model uses a Mixture of Experts (MoE) architecture with 896 routed experts and two shared experts. To optimize performance, tokens are projected into a compressed latent space before reaching the routed experts, where only 16 are activated per token. A custom SiTU-GLU activation function prevents activation explosions during these matrix operations.
