A custom homelab rig built with three decommissioned Nvidia Tesla V100 16GB GPUs achieved over 100 tokens per second when running the Qwen 3.6 35B model at 4K context. The system, which cost $1,042 to assemble, utilizes a Supermicro X10DRG-Q motherboard with two Intel Xeon E5-2687W 12-core CPUs and a Corsair HX1500i power supply.

The testing involved over 130 LLM configurations using llama.cpp, including models like Gemma 4 and Qwen 2.5. At 128K context, the V100 setup maintained 72.7 tokens per second, surpassing the 70 tokens per second threshold for real-time interaction. The builder noted that while adding more cards increases available memory for larger models, it does not necessarily improve inference speed.

Compared to modern hardware, the V100 rig outperformed an M5 Max Mac Studio with 40 GPU cores and 128GB of memory at 64K and 128K context sizes, leading by 19% at the longest context. The setup also consumed approximately 0.02 kWh per 128K 35B run, proving more energy-efficient than older P100 or K80 GPU configurations tested in the same environment.

The builder plans to further optimize the setup by testing different inference backends and implementing speculative decoding. Additionally, a custom-designed cooling solution for the enterprise GPUs is currently in development, with beta testing expected to begin soon for members of the project's mailing list.