Distributed training of foundation models via $\texttt{DDP}$ is limited by interconnect bandwidth. While infrequent communication strategies reduce synchronization frequency, they remain bottlenecked by the memory and communication requirements of optimizer states. Low-rank optimizers can alleviate these constraints; however, in the local-update regime, workers lack access to the full-batch gradients required to compute low-rank projections, which degrades performance. We propose $\texttt{LoRDO}$, a principled framework unifying low-rank optimization with infrequent synchronization. We first demonstrate that, while global projections based on pseudo-gradients are theoretically superior, they permanently restrict the optimization trajectory to a low-rank subspace. To restore subspace exploration, we introduce a full-rank quasi-hyperbolic update. $\texttt{LoRDO}$ achieves near-parity with low-rank $\texttt{DDP}$ in language modeling and downstream tasks at model scales of $125$M--$720$M, while reducing communication by $\approx 10 \times$. Finally, we show that $\texttt{LoRDO}$ improves performance even more in very low-memory settings with small rank/batch size.

翻译：通过 $\texttt{DDP}$ 进行基础模型的分布式训练受限于互连带宽。虽然低频通信策略降低了同步频率，但它们仍然受到优化器状态的内存和通信需求的瓶颈制约。低秩优化器可以缓解这些限制；然而，在本地更新机制下，工作节点缺乏计算低秩投影所需的完整批次梯度，这会降低性能。我们提出了 $\texttt{LoRDO}$，这是一个将低秩优化与低频同步统一起来的原理性框架。我们首先证明，虽然基于伪梯度的全局投影在理论上更优，但它们会永久地将优化轨迹限制在一个低秩子空间中。为了恢复子空间探索，我们引入了全秩拟双曲更新。$\texttt{LoRDO}$ 在 125M--720M 模型规模的语言建模和下游任务中，实现了与低秩 $\texttt{DDP}$ 接近的性能，同时将通信量减少了约 $10 \times$。最后，我们表明，在秩/批次大小较小的极低内存设置中，$\texttt{LoRDO}$ 能进一步提升性能。