Agentic reinforcement learning (RL) is reshaping LLM post-training, but end-to-end training time is dominated by compute-intensive, multi-turn rollouts whose resource demand varies significantly across training steps. Resource-fixed systems cannot adapt to this variation, while resource-elastic approaches that provision external GPUs on demand suffer from high allocation overhead and limited availability. We observe that serving clusters leave substantial GPU compute and memory idle, and propose cooperative elasticity: sharing already-deployed serving GPUs with rollout workloads to provide on-demand elastic capacity. Realizing this is non-trivial, as it must preserve serving SLOs under bursty traffic while minimizing cross-cluster communication overhead. We present ROSE, a system that realizes cooperative elasticity for agentic RL post-training, comprising three components: (1) an SLO-safe co-serving executor that co-locates heterogeneous serving and rollout models on the same GPUs, dynamically sharing memory and compute while preserving serving SLOs; (2) a cross-cluster weight transfer engine that leverages shard-aware routing and weight sparsity for fast synchronization; and (3) an elastic rollout scheduler that dynamically routes rollouts across dedicated and opportunistic serving GPUs. Experiments across multiple model sizes and cluster scales show that ROSE improves end-to-end throughput by 1.3 - 3.3 x over resource-fixed baselines and reduces rollout time by 1.2 - 1.5 x over resource-elastic baselines, with no serving SLO violations.

翻译：智能体强化学习正在重塑大语言模型的后训练过程，但端到端训练时间受限于计算密集型、多轮交互的推演阶段，其资源需求在不同训练步骤间波动显著。固定资源系统难以适应这种变化，而按需调配外部GPU的弹性资源方案则面临高分配开销和可用性受限的问题。我们观察到服务集群中存在大量闲置的GPU计算与内存资源，据此提出协作弹性概念：共享已部署的服务GPU来承载推演负载，提供按需弹性能力。实现这一目标存在诸多挑战，既要应对突发流量仍保证服务SLO，又要最小化跨集群通信开销。本文提出ROSE系统，面向智能体强化学习后训练实现协作弹性机制，包含三大组件：（1）SLO安全的协同执行器，将异构的服务模型与推演模型共置于同一GPU，在保障服务SLO前提下动态共享内存与计算资源；（2）跨集群权重传输引擎，利用分片感知路由与权重稀疏性实现快速同步；（3）弹性推演调度器，在专用GPU与机会式服务GPU间动态路由推演任务。跨多种模型规模与集群规模的实验表明，相较固定资源基线方案，ROSE提升端到端吞吐量1.3-3.3倍；相较弹性资源基线方案，减少推演时间1.2-1.5倍，且未违反任何服务SLO。