Reinforcement learning (RL) is a critical stage in post-training large language models (LLMs), involving repeated interaction between rollout generation, reward evaluation, and centralized learning. Distributing rollout execution offers opportunities to leverage more cost-efficient inference resources, but introduces challenges in wide-area coordination and policy dissemination. We present ECHO-2, a distributed RL framework for post-training with remote inference workers and non-negligible dissemination latency. ECHO-2 combines centralized learning with distributed rollouts and treats bounded policy staleness as a user-controlled parameter, enabling rollout generation, dissemination, and training to overlap. We introduce an overlap-based capacity model that relates training time, dissemination latency, and rollout throughput, yielding a practical provisioning rule for sustaining learner utilization. To mitigate dissemination bottlenecks and lower cost, ECHO-2 employs peer-assisted pipelined broadcast and cost-aware activation of heterogeneous workers. Experiments on GRPO post-training of LLMs ranging from 4B to 32B parameters under real wide-area bandwidth regimes show that ECHO-2 significantly improves cost efficiency while preserving RL reward comparable to strong baselines.

翻译：强化学习（RL）是大语言模型（LLM）后训练的关键阶段，涉及展开生成、奖励评估与集中学习之间的反复交互。分布式展开执行提供了利用更具成本效益的推理资源的机遇，但引入了广域协调与策略传播挑战。本文提出ECHO-2，一种面向后训练的分布式RL框架，支持远程推理工作节点并承受不可忽略的传播延迟。ECHO-2将集中式学习与分布式展开相结合，将有界策略过时性作为用户可控参数，使展开生成、传播与训练三者重叠执行。我们提出基于重叠的容量模型，关联训练时间、传播延迟与展开吞吐量，给出维持学习器利用率的实用配置规则。为缓解传播瓶颈并降低成本，ECHO-2采用对等辅助流水线广播与成本感知异构工作节点激活机制。在4B至32B参数规模LLM的GRPO后训练实验中，基于真实广域带宽环境，ECHO-2在保持与强基线相当RL奖励的同时，显著提升了成本效率。