Distributing LLM inference across geographical regions can improve Time-to-First-Token (TTFT) by regionalizing service deployments. While existing multi-region load balancers save prefill computation by prioritizing Key--Value (KV) Cache hit rate, they ignore cluster networking latency, a critical factor in routing decisions. We introduce GORGO, a method for minimizing TTFT by optimizing a total serving cost as a function of available compute, network latency, and prefix caching. Using extensive profiling on custom infrastructure, we analyze component-level latency bottlenecks and benchmark GORGO against three baselines: (1) naive least-load routing, which ignores prefix-cache overlap; (2) prefix-similarity routing, which selectively pushes requests to the replica with the highest cached-prefix overlap; and (3) a centralized HTTP proxy that runs the GORGO policy while tracking requests across all nodes. We demonstrate that GORGO reduces P99 TTFT through network-aware routing and improves average TTFT by preventing pathological cross-region forwarding. Additionally, we find that GORGO-proxy overcomes synchronization overhead in previous methods and is 2.5x faster on median TTFT, demonstrating the success of a centralized router.

翻译：通过区域化服务部署，将大语言模型推理分布到不同地理区域可以改善首词生成时间。现有的多区域负载均衡器虽通过优先考虑键值缓存命中率来节省预填充计算，却忽略了集群网络延迟这一路由决策的关键因素。我们提出GORGO方法，通过将可用计算资源、网络延迟和前缀缓存作为变量构建总服务成本函数并对其进行优化，以实现最小化首词生成时间的目标。基于定制基础设施的广泛性能剖析，我们分析了组件级延迟瓶颈，并将GORGO与三种基线方案进行对比：(1) 忽略前缀缓存重叠的简单最小负载路由；(2) 将请求选择性推送至缓存前缀重叠度最高副本的前缀相似性路由；(3) 在所有节点上跟踪请求并执行GORGO策略的集中式HTTP代理。实验表明，GORGO通过感知网络状态的路由机制降低了P99首词生成时间，并通过避免异常的跨区域转发改善了平均首词生成时间。此外，我们发现GORGO代理克服了先前方法的同步开销，其中位首词生成时间提升2.5倍，这验证了集中式路由架构的有效性。