Data centers capable of running large language models (LLMs) are spread across the globe. Some have high end GPUs for running the most advanced models (100B+ parameters), and others are only suitable for smaller models (1B parameters). The most capable GPUs are under high demand thanks to the rapidly expanding applications of LLMs. Choosing the right location to run an LLM inference workload can have consequences on the latency of requests due to these high demands. In this work, we explore options to shift some aspects of inference to the under-utilized data centers. We first observe the varying delays affecting inference in AWS services from different regions, demonstrating that load is not spread evenly. We then introduce WANSpec, which offloads part of LLM generation to the under-utilized data centers. In doing so, WANSpec can mitigate capacity issues as well as effectively use on-site compute (ie at universities) to augment cloud providers. This is done with speculative decoding, a widely used technique to speed up auto-regressive decoding, by moving the draft model to the under-utilized compute resources. Our experiments in simulation and cloud deployments show that WANSpec can judiciously employ redundancy to avoid increases in latency while still reducing the forward passes of speculative decoding's draft model in high demand data centers by over 50%.

翻译：能够运行大语言模型（LLM）的数据中心遍布全球。其中一些配备高端GPU，可运行最先进的模型（参数规模达1000亿以上），而另一些仅适用于较小模型（10亿参数）。由于LLM应用的迅速扩展，性能最强的GPU需求旺盛。鉴于这种高需求，选择合适的位置运行LLM推理工作负载会对请求延迟产生显著影响。本研究探索将推理的某些环节转移至利用率不足的数据中心的方案。我们首先观测了影响不同区域AWS服务中推理的延迟变化，证明负载分布并不均衡。随后提出WANSpec系统，该系统将部分LLM生成任务卸载至利用率不足的数据中心。通过这种方式，WANSpec既能缓解算力紧张问题，又能有效利用本地计算资源（如高校设施）来增强云服务商能力。其实现基于推测解码技术——一种广泛用于加速自回归解码的方法——通过将草稿模型部署至利用率不足的计算资源。我们在仿真环境与云平台部署中的实验表明，WANSpec能通过智能冗余策略避免延迟增加，同时在高需求数据中心将推测解码草稿模型的前向传播次数降低超过50%。