Round-Robin’s Impact on KV-Cache and LLM Inference Latency Revealed

The Critical Role of Load Balancing in LLM Performance

Recent measurements by Red Hat have uncovered significant performance discrepancies in large language model (LLM) inference when utilizing different load balancing methods, despite similar requests per second (RPS) metrics. The study demonstrated that while both traditional round-robin and prefix-aware routing exhibited nearly identical RPS values (0.52 and 0.49 respectively), the p95 time to first token (TTFT) varied drastically. Round-robin showed a p95 TTFT of 745 ms, which is 2.7 times worse than the 272 ms recorded for prefix-aware routing.

KV-Cache: A Key to Efficient Routing

The core reason behind this performance gap lies in the utilization of the KV-cache. In the experiments, the GPUs, model, and overall workload remained constant, with only the replica selection method changing. The standard RPS counter fails to account for the internal processing a pod undertakes for a specific request. If a request is routed to a replica where the necessary prefix is already present in the KV-cache, the model can commence token generation much faster. Conversely, if the same request lands on an adjacent replica without this pre-cached state, it must re-compute the entire query head, leading to substantial delays.

Applicability and Infrastructure Considerations

This analysis is particularly relevant for engineers deploying self-hosted LLM inference in Kubernetes across multiple GPUs, as well as for platform owners who assess workload based on GPU utilization and request counts. For single-replica setups, routing challenges do not arise, but they become critical at scale. All presented examples are based on public measurements from various teams, focusing on the infrastructure impact of routing on GPU usage, latency, and Kubernetes cluster requirements. VK Tech provides GPU nodes and vGPU solutions for deploying such high-performance systems.