Red Hat conducted a performance test comparing two load balancing methods for chat traffic distributed across four NVIDIA L4 GPUs. While the request per second (RPS) metrics remained nearly identical at 0.52 for round-robin and 0.49 for prefix-aware routing, the latency results showed a significant discrepancy.
The p95 time to first token (TTFT) for the round-robin approach reached 745 ms, whereas prefix-aware routing achieved 272 ms. This 2.7-fold difference highlights how standard load balancing fails to account for the state of the KV-cache, forcing replicas to recompute request headers when data is missing.
The analysis emphasizes that RPS metrics alone are insufficient for monitoring self-hosted LLM inference in Kubernetes. Infrastructure engineers must consider how replica selection strategies impact GPU utilization and overall system latency, particularly when scaling models across multiple hardware units.
