#llamacpp

Running 35B MoE Models on Heterogeneous GPU Setups

A test of a 35B MoE model on a mixed setup of RTX 5080 and RTX 3060 GPUs demonstrates how layer-wise splitting mitigates PCIe bandwidth limitations.