Running Qwen3.8-Flash-Next 125B on Consumer GPUs

The Qwen3.8-Flash-Next model, based on a 125B Mixture-of-Experts (MoE) backbone, features 51B n-gram embeddings, native vision capabilities, and a 262K token context window. The model's full checkpoint in bfloat16 format occupies approximately 360 GB on Hugging Face, with the n-gram embedding table alone taking up about 102 GB.

To enable local execution on consumer hardware, AirLLM utilizes layer-wise inference and memory mapping (mmap) for the n-gram table. This approach allows the model to run on a GPU with as little as 8 GB of VRAM and a system with 64 GB of RAM, as the decoder layers are streamed sequentially from the disk and the n-gram table remains on the SSD.

In a test conducted on a single RTX 4090, the peak VRAM consumption was 5.95 GB without using 4-bit quantization or distillation. The setup requires a high-speed SSD with several hundred gigabytes of free space to store both the original checkpoint and its layer-wise split version during the initial launch.

The model is registered as Qwen4ExpForConditionalGeneration in Hugging Face and is integrated into the transformers library. It is designed for coding agents, computer use, and processing documents and screenshots via native vision. AirLLM is distributed as open source under the MIT license.