The llama-modchip project, released under the Apache 2.0 license, provides a set of patches for ggml and llama.cpp kernels, quantization tools, and scripts. This software enables the execution of sparse Mixture-of-Experts (MoE) models directly on a CPU, bypassing the need for a dedicated GPU.
The DeepSeek-V4-Flash model, which features 284 billion parameters with 13 billion active parameters, was tested on a machine equipped with a 16-core Ryzen 9 9955HX processor and 64 GB of RAM. The setup achieved a prefill speed of 44 tokens per second and a generation speed ranging from 9 to 15 tokens per second.
Memory usage peaked at approximately 58 GB with a context window of 256,000 tokens. The project utilizes aggressive quantization of routed experts down to 1.56 bits per weight using the IQ1_S scheme, allowing the model to fit within the available 64 GB of system memory without using swap space.
The developer notes that the model's logits match those of an unpatched build using identical quantized weights. However, these custom quantized files are currently incompatible with standard llama.cpp, Ollama, or LM Studio, requiring the specific project build to function.
