Tsinghua University has open-sourced AKV, a cache engineering server designed for LLM agents on SGLang. The system allows agents to manage their KV cache by dropping or repositioning entries, which avoids the need for recomputation. KV cache reads account for 82.4% of total API costs, making this optimization significant for performance.
AKV introduces three operations: Append for new tokens, Drop for removing entries, and Repos for rewriting keys using RoPE rotation. The server extends the SGLang radix tree to handle these operations and uses segment-batch attention to restore evicted states. The interface is stateless and compatible with OpenAI standards.
In testing with the GPT-OSS-120B model, AKV increased the BrowseComp-Plus pass rate from 55% to 62%. Compared to text engineering methods, AKV reduced costs by 21.9%. At a concurrency of 16, the system achieved a P95 latency of 30.63 seconds, significantly faster than the 229.90 seconds recorded by the standard baseline.
The system supports built-in policies such as turn-based, token-based, or model-based strategies. A CacheTool allows models to trigger context drops independently. On 100 SWE-bench Verified tasks, AKV achieved a 47% pass rate, outperforming the standard method at 40.7% and text engineering at 32.3%.
