Large language models have demonstrated extraordinary performance in many AI tasks but are expensive to use, even after training, due to their requirement of high-end GPUs. Recently, a distributed system called PETALS was developed to lower the barrier for deploying LLMs by splitting the model blocks across multiple servers with low-end GPUs distributed over the Internet, which was much faster than swapping the model parameters between the GPU memory and other cheaper but slower local storage media. However, the performance of such a distributed system critically depends on the resource allocation, and how to do so optimally remains unknown. In this work, we present the first systematic study of the resource allocation problem in distributed LLM inference, with focus on two important decisions: block placement and request routing. Our main results include: experimentally validated performance models that can predict the inference performance under given block placement and request routing decisions, a formulation of the offline optimization of block placement and request routing as a mixed integer linear programming problem together with the NP-hardness proof and a polynomial-complexity algorithm with guaranteed performance, and an adaptation of the offline algorithm for the online setting with the same performance guarantee under bounded load. Through both experiments and experimentally-validated simulations, we have verified that the proposed solution can substantially reduce the inference time compared to the state-of-the-art solution in diverse settings with geographically-distributed servers. As a byproduct, we have also developed a light-weighted CPU-only simulator capable of predicting the performance of distributed LLM inference on GPU servers, which can evaluate large deployments and facilitate future research for researchers with limited GPU access.

翻译：大型语言模型在许多人工智能任务中展现出卓越性能，但由于需要高端GPU，即使在训练完成后使用成本依然高昂。近期开发的分布式系统PETALS通过将模型块分配到互联网上多台配备低端GPU的服务器，显著降低了部署大型语言模型的壁垒，其速度远高于在GPU内存与其他更廉价但速度较慢的本地存储介质之间交换模型参数。然而，此类分布式系统的性能关键取决于资源分配策略，而如何实现最优分配仍属未知。本研究首次系统性地探讨了分布式大型语言模型推理中的资源分配问题，重点关注两个关键决策：块放置与请求路由。主要成果包括：经实验验证的性能模型（可预测给定块放置与请求路由决策下的推理性能）、将块放置与请求路由的离线优化表述为混合整数线性规划问题并附NP难性证明及具有性能保证的多项式复杂度算法、以及适用于在线场景的离线算法改进版本（在有限负载下保持相同性能保证）。通过实验与经实验验证的仿真，我们证实所提方案相比现有最优方案，在不同地理分布式服务器配置中能显著降低推理时间。作为副产品，我们还开发了一个轻量级纯CPU模拟器，能够预测GPU服务器上分布式大型语言模型推理的性能，该工具可评估大规模部署场景，并为GPU资源有限的研究者开展后续研究提供便利。