Tantor Labs and Botsman Platform, both part of Astra Group, have unveiled a hardware-software solution for offloading KV-cache from GPU HBM memory to external NVMe storage. The system utilizes a specialized hardware engine to manage data access via RDMA, bypassing the need for DRAM-based caching.
The technology integrates with vLLM, LMCache, and Nvidia Dynamo. It supports a wide range of GPUs, including Nvidia Ampere, Ada Lovelace, Hopper, and Blackwell, as well as AMD Instinct and Radeon, and chips from Alibaba, MetaX, and Baidu. The architecture employs 400 GbE network adapters and NVMe-oF protocols.
By enabling compute-KV storage disaggregation, the solution allows context memory to be shared across multiple GPU servers. This approach decouples memory management from specific accelerators, facilitating independent scaling of AI infrastructure and reducing the need for additional GPU hardware.
Internal tests demonstrated that the hardware module allows serving four times as many clients on the same GPU capacity while maintaining latency requirements. This results in lower costs per request and per token, as the system provides NVMe-level storage capacity at speeds comparable to DRAM.
