Pretraining language models with extended context windows enhances their ability to leverage rich information during generation. Existing methods split input sequences into chunks, broadcast them across multiple devices, and compute attention block by block which incurs significant communication overhead. While feasible in high-speed clusters, these methods are impractical for decentralized training over low-bandwidth connections. We propose a compression method for communication-efficient context parallelism in decentralized settings, achieving a remarkable compression rate of over 95\% with negligible overhead and no loss in convergence. Our key insight is to exploit the intrinsic low-rank structure of activation outputs by dynamically constraining them to learned mixtures of subspaces via efficient reparameterizations. We demonstrate scaling billion-parameter decentralized models to context lengths exceeding 100K tokens on networks as slow as 300Mbps, matching the wall-clock convergence speed of centralized models on 100Gbps interconnects.

翻译：扩展上下文窗口的预训练语言模型能够增强其在生成过程中利用丰富信息的能力。现有方法将输入序列分块处理，通过多设备广播并逐块计算注意力机制，导致显著的通信开销。虽然该方法在高速集群中可行，但在低带宽连接的分散训练场景中不具实用性。我们提出一种面向分散化场景的通信高效上下文并行压缩方法，在几乎零开销且不影响收敛性的前提下实现了超过95%的显著压缩率。核心思路在于利用激活输出的内在低秩结构，通过高效重参数化方法将其动态约束至学习得到的子空间混合体中。实验证明，我们能在低至300Mbps的网络上将数十亿参数分散模型扩展到超过10万令牌的上下文长度，其实际收敛速度与在100Gbps互连上的集中式模型持平。