This work presents a new approach to decentralized training-SeedFlood-designed to scale for large models across complex network topologies and achieve global consensus with minimal communication overhead. Traditional gossip-based methods suffer from message communication costs that grow with model size, while information decay over network hops renders global consensus inefficient. SeedFlood departs from these practices by exploiting the seed-reconstructible structure of zeroth-order updates and effectively making the messages near-zero in size, allowing them to be flooded to every client in the network. This mechanism makes communication overhead negligible and independent of model size, removing the primary scalability bottleneck in decentralized training. Consequently, SeedFlood enables training in regimes previously considered impractical, such as billion-parameter models distributed across hundreds of clients. Our experiments on decentralized LLM fine-tuning demonstrate thatSeedFlood consistently outperforms gossip-based baselines in both generalization performance and communication efficiency, and even achieves results comparable to first-order methods in large scale settings.

翻译：本文提出一种去中心化训练新方法——SeedFlood，该方法旨在支持复杂网络拓扑下的大规模模型训练，并以最小通信开销实现全局共识。传统基于Gossip的方法存在消息通信成本随模型规模增长的问题，且信息在网络跳转中的衰减导致全局共识效率低下。SeedFlood通过利用零阶更新的种子可重构特性，使消息尺寸趋近于零，从而能够将消息洪泛至网络中所有客户端。该机制使通信开销可忽略且与模型规模无关，消除了去中心化训练中的主要可扩展性瓶颈。因此，SeedFlood使得在先前被认为不可行的场景中进行训练成为可能，例如在数百个客户端上分布式训练百亿参数模型。我们在去中心化大语言模型微调实验中发现，SeedFlood在泛化性能和通信效率上均持续优于基于Gossip的基线方法，甚至在大规模设置中取得了与一阶方法相当的结果。