The author built a local server for coding agents using an Intel Core i7-4790, 32 GB of DDR3 RAM, and two NVIDIA RTX 3060 12 GB graphics cards. The goal was to run a Qwen3.8-27B model with a 128K context window while avoiding cloud subscription fees.
Initial tests with a standard layer-based distribution resulted in speeds of approximately 9 tokens per second on a long context. By switching to tensor parallelism, the author achieved nearly double the performance, reaching 16.09 tokens per second at 120K context.
The experiment utilized llama.cpp build b10991. The author found that using Q4 quantization for the KV-cache was necessary to prevent CUDA out-of-memory errors during long context prefill, ensuring stability for a 128K window.
Testing with speculative decoding via Multi Token Prediction (MTP) showed high speeds on short requests, but the memory overhead forced a reduction in the context window to 96K. Additionally, using NCCL for multi-GPU communication proved slower than the default mechanism on this specific hardware topology.
The final setup, optimized for daily use, reached an average of 39.6 tokens per second in real-world OpenCode sessions. The author concluded that tensor parallelism provided the most significant performance gain for this specific multi-GPU configuration.
