Hangzhou DeepSeek Artificial Intelligence Basic Technology Research has open-sourced DeepInfer, an inference framework used in its production services. The software is available on GitHub under the Apache 2.0 license.
DeepInfer currently supports the DeepSeek-V4.1-Flash model. It is compatible with Nvidia SM100 Blackwell GPUs and Huawei Ascend 950 NPUs. Features for output consistency and performance benchmarks are currently limited to Nvidia hardware.
The framework utilizes DSpark for speculative decoding, which reduces draft token checks by approximately 30% and provides a 4% to 7% throughput gain. Another technique reduces prefill computation by nearly half by processing only the final 128 tokens.
DeepSeek notes that key-value cache management and load-balancing code remain internal. Efficient deployments require a minimum of 64 GPUs for prefill and 64 for decoding. The project is based on an early fork of vLLM.
The company plans to deploy at least 160,000 Ascend 950DT accelerators in Inner Mongolia. This release follows a U.S. Department of Homeland Security directive requiring federal agencies to remove DeepSeek products.
