The author tested several MoE models on a GTX 1080 Ti using llama-server. The setup included the qwen4exp architecture support and a WebUI for real-time parameter management. The server was configured with specific flags, including --tools all for agentic tasks and --jinja for chat templates, while limiting parallel requests to one.
Tested models included the 120B gpt-oss-120b in Q8_K_XL quantization, two versions of the 125B Qwen3.8-Flash-Next (Q3_K_XL and Q4_K_XL), and the 35B Qwen3.8-35B-A3B in Q4_K_M quantization. The 35B model occupies approximately 21 GB, allowing for partial offloading to RAM.
The 35B model outperformed larger alternatives in coding tasks. It achieved prompt processing speeds of up to 400 tokens/s and generation speeds of up to 30 tokens/s. Increasing the context size from 32K to 128K resulted in a performance drop of only 10–15%.
In comparison, the 120B model struggled with long-context errors, while the 125B models were significantly slower, with generation speeds peaking at 11 tokens/s. The 35B model demonstrated superior instruction following and code generation capabilities.
