Red Hat conducted a performance test comparing two load balancing methods for chat traffic distributed across four NVIDIA L4 GPUs. The study evaluated standard round-robin routing against prefix-aware routing while maintaining identical model and hardware configurations for both test runs.
The test results showed that while the requests per second (RPS) remained similar at 0.52 for round-robin and 0.49 for prefix-aware routing, the latency metrics differed significantly. The p95 time to first token (TTFT) was 745 ms for round-robin, compared to 272 ms for prefix-aware routing, marking a 2.7-fold difference.
The discrepancy arises because standard round-robin does not account for the state of the KV-cache. When a request is routed to a replica lacking the necessary prefix in its cache, the system must recompute the request head, which increases latency. Prefix-aware routing optimizes this by leveraging existing cache states.
This analysis highlights the limitations of using RPS as a primary metric for infrastructure performance, as it fails to capture the actual computational load required for specific requests. Engineers deploying self-hosted LLM inference in Kubernetes environments can use these findings to optimize replica selection and improve overall system responsiveness.
