The ExVRAM Lab research project has successfully demonstrated the execution of a 27-billion parameter Qwen3.8–27B model entirely on an NVIDIA GeForce RTX 5060 8 GB graphics card. By utilizing the UD-IQ2_XXS quantization format within the llama.cpp framework, the team achieved a median generation speed of 30.10 tokens per second.
The configuration ensures that all 65 layers of the model are stored on the GPU, eliminating the need for decoder weight offloading to system RAM. This setup requires 7767 MiB of VRAM, leaving minimal space for KV cache and runtime buffers. The researchers utilized q4_0 for the KV cache, which provided the most stable performance compared to other tested formats.
The project emphasizes the importance of rigorous benchmarking methodology. Initial tests yielded misleadingly high speeds due to underutilized context windows. The team now distinguishes between configured context size and actual token occupancy to ensure results reflect real-world performance under heavy load.
Current efforts are focused on the P9 phase, which evaluates the quality of the model after aggressive quantization. Researchers are testing reasoning, coding, and instruction-following capabilities to determine if the model maintains its utility. Future experiments include exploring sub-two-bit quantization formats to further reduce memory footprint.
