Large language models (LLMs) are useful in many NLP tasks and become more capable with size, with the best open-source models having over 50 billion parameters. However, using these 50B+ models requires high-end hardware, making them inaccessible to most researchers. In this work, we investigate methods for cost-efficient inference and fine-tuning of LLMs, comparing local and distributed strategies. We observe that a large enough model (50B+) can run efficiently even on geodistributed devices in a consumer-grade network. This could allow running LLM efficiently by pooling together idle compute resources of multiple research groups and volunteers. We address two open problems: (1) how to perform inference and fine-tuning reliably if any device can disconnect abruptly and (2) how to partition LLMs between devices with uneven hardware, joining and leaving at will. In order to do that, we develop special fault-tolerant inference algorithms and load-balancing protocols that automatically assign devices to maximize the total system throughput. We showcase these algorithms in Petals - a decentralized system that runs Llama 2 (70B) and BLOOM (176B) over the Internet up to 10x faster than offloading for interactive generation. We evaluate the performance of our system in simulated conditions and a real-world setup spanning two continents.

翻译：大型语言模型（LLM）在许多自然语言处理任务中具有实用价值，且其能力随规模扩大而增强，目前最佳开源模型已拥有超过500亿参数。然而使用这些500B+模型需要高端硬件，导致多数研究者难以获取。本文研究了LLM在成本效益推理与微调方面的策略，对比了本地与分布式方案。我们观察到，足够大的模型（500B+）甚至能在消费级网络中的地理分布式设备上高效运行。这有望通过汇集多个研究团队和志愿者的闲置计算资源来高效运行LLM。我们解决了两个开放性问题：（1）如何在任意设备可能突然断连的情况下可靠执行推理与微调；（2）如何在硬件配置不均、可随时加入或退出的设备间切分LLM。为此我们开发了特殊的容错推理算法和负载均衡协议，可自动分配设备以最大化系统总吞吐量。我们在Petals系统中展示了这些算法——该去中心化系统在互联网上运行Llama 2（70B）和BLOOM（176B），交互式生成速度比卸载方式快达10倍。我们在模拟环境及横跨两大洲的实际部署中评估了系统性能。