Many NLP tasks benefit from using large language models (LLMs) that often have more than 100 billion parameters. With the release of BLOOM-176B and OPT-175B, everyone can download pretrained models of this scale. Still, using these models requires high-end hardware unavailable to many researchers. In some cases, LLMs can be used more affordably via RAM offloading or hosted APIs. However, these techniques have innate limitations: offloading is too slow for interactive inference, while APIs are not flexible enough for research that requires access to weights, attention or logits. In this work, we propose Petals - a system for inference and fine-tuning of large models collaboratively by joining the resources of multiple parties. We demonstrate that this strategy outperforms offloading for very large models, running inference of BLOOM-176B on consumer GPUs with $\approx$ 1 step per second, which is enough for many interactive LLM applications. Unlike most inference APIs, Petals also natively exposes hidden states of served models, allowing to train and share custom model extensions based on efficient fine-tuning methods.

翻译：摘要：许多自然语言处理任务受益于使用参数量常超千亿的大型语言模型。随着BLOOM-176B和OPT-175B的发布，研究者可下载此规模的预训练模型。然而，使用这些模型需高端硬件，而许多研究者无法获取。在某些情况下，可通过RAM卸载或托管API以较低成本使用大型语言模型，但这些技术存在固有局限：卸载方式对交互式推理而言速度过慢，而API则缺乏研究所需的灵活性（如访问模型权重、注意力机制或对数概率）。本文提出Petals——一种通过多方资源协作实现大规模模型推理与微调的系统。我们证明，该策略在超大规模模型上优于卸载方案：使用消费级GPU运行BLOOM-176B推理时，速度可达约每秒1步，足以满足多数交互式大型语言模型应用需求。与多数推理API不同，Petals还原生暴露所服务模型的隐藏状态，允许基于高效微调方法训练和共享定制模型扩展。