The author tested the Qwen3.6-35B-A3B MoE model using a heterogeneous setup with an RTX 5080 (16 GB) and an RTX 3060 (12 GB). The RTX 3060 was connected via a PCIe Gen2 x4 slot, providing a bandwidth of 2 GB/s, while the RTX 5080 utilized PCIe Gen3 x16.
The system achieved a generation speed of 109 tokens per second for short prompts and 77 tokens per second with a 53,000-token context. The setup uses layer-wise splitting, where the model layers are distributed between the two GPUs, and only activations are transferred over the PCIe bus.
The author notes that the MoE architecture, which activates only about 3B parameters out of 35B per token, minimizes data transfer requirements. This prevents the slow PCIe Gen2 x4 bus from becoming a bottleneck, as the weights remain on their respective cards.
Performance analysis showed that the RTX 3060 operated at 62% load, while the RTX 5080 remained at 15% load. The primary bottleneck is the sequential dependency between the cards, as they process layers one after another, leading to increased latency compared to a single-card setup.
The configuration utilized llama.cpp with KV-cache quantization to q8_0 and mmap loading. The author provided a preset repository on GitHub for setting up similar environments, supporting various backends including CUDA.
