Distributed Low-Communication (DiLoCo) training reduces communication overhead by allowing workers to perform multiple local optimization steps before sending pseudo-gradients to a global outer update. Its asynchronous variant further improves hardware utilization by removing synchronization barriers, but at the cost of stale pseudo-gradients computed from outdated model states. As a result, these updates can become misaligned with the current global optimization direction, particularly in heterogeneous systems. This issue becomes even more pronounced when data are non-IID, a setting that has not been well studied in asynchronous low-communication training. To address this limitation, we propose \textbf{HeLoCo}, a direction-aware correction method for asynchronous low-communication training that uses outer momentum as a reference for the current optimization trajectory and selectively adjusts incoming pseudo-gradients before the outer update. Updates that remain aligned are preserved, while directionally conflicting components are corrected. On multilingual language-model training with heterogeneous workers and non-IID data, HeLoCo consistently improves validation loss. It outperforms existing asynchronous DiLoCo-based baselines by up to 7.5\% at a fixed token budget, exceeds asynchronous momentum look-ahead by up to 3.3\% at a fixed wall-clock budget, and surpasses the synchronous baseline by up to 22.1\% under severe system heterogeneity. Our analysis further shows how staleness, worker speed, and data heterogeneity shape update quality and convergence in highly decentralized and heterogeneous training setups.

翻译：分布式低通信（DiLoCo）训练通过允许工作节点在向全局外层更新发送伪梯度前执行多次本地优化步骤来降低通信开销。其异步变体通过移除同步屏障进一步提升了硬件利用率，但代价是伪梯度可能基于过时的模型状态计算而产生陈旧性。这些更新可能与当前全局优化方向产生偏差，尤其是在异构系统中。当数据呈非独立同分布（非IID）时，这一问题更为突出，而此设定在异步低通信训练中尚未得到充分研究。为解决这一局限，我们提出**HeLoCo**——一种面向异步低通信训练的定向感知校正方法，该方法以外层动量为参考捕捉当前优化轨迹，并在外层更新前对传入的伪梯度进行选择性调整：保留方向一致的更新分量，校正方向冲突的更新分量。在异构工作节点与非IID数据下的多语言语言模型训练中，HeLoCo持续改善验证损失。在固定词元预算下，其性能超越现有基于DiLoCo的异步基线方法最高达7.5%；在固定总时间预算下，超越异步动量前瞻方法最高达3.3%；在严重系统异构条件下，超越同步基线方法最高达22.1%。我们的分析进一步揭示了在高度去中心化与异构训练场景中，陈旧性、工作节点速度与数据异构性如何影响更新质量与收敛性。