Decentralized LLM inference distributes computation among heterogeneous nodes across the internet, offering a performant and cost-efficient solution, alternative to traditional centralized inference. However, the low cross-node network bandwidth makes communication the primary bottleneck. In this paper, we introduce BloomBee, an internet-scale distributed LLM inference framework. BloomBee integrates LLM-layer assignment, micro-batching and tensor offloading to optimize communication from multiple dimensions. Additionally, BloomBee formulates the coordination of these techniques as an optimization problem and solves it using dynamic programming. BloomBee also customizes lossless compression and speculative decoding according to low-bandwidth network settings to reduce communication overhead. We evaluate BloomBee across a spectrum of network environments and show that it improves service throughput by up to 1.76x. It also reduces average latency by up to 43.20% compared to state-of-the-art decentralized LLM inference systems. BloomBee is open-sourced.

翻译：去中心化的大语言模型推理通过在互联网上的异构节点间分布计算，提供了一种性能优越且成本高效的解决方案，可作为传统集中式推理的替代方案。然而，跨节点网络带宽较低使得通信成为主要瓶颈。本文介绍BloomBee，一个互联网规模的分布式大语言模型推理框架。BloomBee整合了LLM层分配、微批处理和张量卸载，从多个维度优化通信。此外，BloomBee将这些技术的协同问题形式化为一个优化问题，并使用动态规划求解。BloomBee还针对低带宽网络环境定制了无损压缩和推测性解码，以减少通信开销。我们在多种网络环境下评估BloomBee，结果显示其服务吞吐量提升高达1.76倍。与当前最先进的去中心化大语言模型推理系统相比，平均延迟降低高达43.20%。BloomBee已开源。