Scaling large language models (LLMs) demands extensive data and computing resources, which are traditionally constrained to data centers by the high-bandwidth requirements of distributed training. Low-bandwidth methods like federated learning (FL) could enable collaborative training of larger models across weakly-connected GPUs if they can effectively be used for pre-training. To achieve this, we introduce Photon, the first complete system for federated end-to-end LLM training, leveraging cross-silo FL for global-scale training with minimal communication overheads. Using Photon, we train the first federated family of decoder-only LLMs from scratch. We show that: (1) Photon can train model sizes up to 7B in a federated fashion while reaching an even better perplexity than centralized pre-training; (2) Photon model training time decreases with available compute, achieving a similar compute-time trade-off to centralized; and (3) Photon outperforms the wall-time of baseline distributed training methods by 35% via communicating 64x-512xless. Our proposal is robust to data heterogeneity and converges twice as fast as previous methods like DiLoCo. This surprising data efficiency stems from a unique approach combining small client batch sizes with extremely high learning rates, enabled by federated averaging's robustness to hyperparameters. Photon thus represents the first economical system for global internet-wide LLM pre-training.

翻译：摘要：规模化大语言模型（LLM）需要海量数据和计算资源，传统上由于分布式训练对高带宽的要求，这些资源局限于数据中心内。如果能够有效用于预训练，联邦学习等低带宽方法可支持在弱连接GPU间协作训练更大规模的模型。为此，我们提出Photon——首个用于联邦端到端LLM训练的系统，利用跨孤岛联邦学习以极低通信开销实现全球规模训练。借助Photon，我们从零训练了首个联邦式解码器专用LLM族。研究表明：（1）Photon能以联邦方式训练高达7B的模型，其困惑度甚至优于集中式预训练；（2）Photon的模型训练时间随可用算力增加而降低，实现了与集中式相似的算力-时间权衡；（3）通过减少64至512倍的通信量，Photon将基准分布式训练方法的耗时显著降低了35%。我们的方案对数据异质性具有鲁棒性，且收敛速度是DiLoCo等先前方法的两倍。这种惊人的数据效率源于独特的方法：结合小型客户端批量与极高学习率，而这得益于联邦平均超参数鲁棒性。因此，Photon代表了首个经济可行的全球互联网规模LLM预训练系统。