Lightbits Labs has launched Inferra, a software engine designed to optimize AI inference by offloading key-value cache data from GPU high-bandwidth memory to DRAM and NVMe storage. The system utilizes predictive prefetching to move data near the GPU before it is required, aiming to reduce idle time and improve resource utilization.
According to company benchmarks, Inferra can increase concurrent session density by more than 16 times and support context windows exceeding 10 million tokens on commodity hardware. Lightbits reports that the engine can reduce the time to the first token by over 100-fold in specific long-context workloads, while achieving cache hit rates of nearly 99.9%.
The software includes quality-of-service controls, data encryption, and multi-tenant isolation. It also supports data migration between GPU clusters. The technology is designed to address capacity and retrieval speed limitations by breaking context into smaller blocks and delivering them to the GPU just in time.
Lightbits is currently conducting production pilots for Inferra, primarily targeting neocloud providers. The company plans to demonstrate the engine at the AI Infra Summit in Santa Clara. The solution is particularly aimed at environments utilizing retrieval-augmented generation, AI agents, and lengthy prompts.
