In large-scale reinforcement learning (RL) systems with decoupled Trainer-Rollout execution, the Trainer must regularly synchronize policy weights to the Rollout side to limit policy staleness. When inter-node bandwidth is abundant, such synchronization is usually only a small fraction of end-to-end cost. As model size grows, however, the communication demand rises rapidly. In bandwidth-constrained or network-variable deployments -- for example, cross-datacenter or cross-cluster settings, heterogeneous resource pools, and online RL -- weight synchronization can become a dominant bottleneck for throughput and tail latency. We observe that, in mainstream large-model RL training, the locations where parameters actually change are highly sparse at the element level (often 99%+ sparsity). Building on this observation, we propose and implement SparseRL-Sync, which replaces full-weight transfers with a lossless sparse update payload (indices and values) that can be exactly reconstructed on the inference side, thereby preserving 100% fidelity. Under a simplified cost model, sparse synchronization reduces the per-update communication volume from S to approximately S/X; with 99% sparsity (X ~ 100), this yields about a 100x reduction in transmitted data. Combined with appropriate bucketing, SparseRL-Sync also reduces launch and control-plane overhead, significantly improving scalability and end-to-end efficiency in bandwidth-limited and highly asynchronous RL settings.

翻译：在采用解耦式训练-推演执行架构的大规模强化学习系统中，训练端必须定期向推演端同步策略权重以限制策略陈旧性。当节点间带宽充裕时，此类同步通常仅占端到端开销的一小部分。然而随着模型规模增长，通信需求急剧上升。在带宽受限或网络波动部署场景下——例如跨数据中心或跨集群环境、异构资源池及在线强化学习——权重同步可能成为吞吐量和尾部延迟的主要瓶颈。我们观察到，主流大规模模型强化学习训练中参数实际发生变更的位置在元素层级高度稀疏（通常可达99%以上稀疏度）。基于此发现，我们提出并实现了SparseRL-Sync，以无损稀疏更新负载（索引与数值）替代完整权重传输，该负载可在推理侧精确重构，从而保持100%保真度。在简化成本模型下，稀疏同步将每次更新的通信量从S缩减至约S/X；当稀疏度为99%（X≈100）时，传输数据量降低约百倍。结合适当的分桶策略，SparseRL-Sync还可降低启动及控制平面开销，显著提升带宽受限与高度异步强化学习场景下的可扩展性与端到端效率。