Reinforcement learning (RL) has become essential for unlocking advanced reasoning capabilities in large language models (LLMs). RL workflows involve interleaving rollout and training stages with fundamentally different resource requirements. Rollout typically dominates overall execution time, yet scales efficiently through multiple independent instances. In contrast, training requires tightly-coupled GPUs with full-mesh communication. Existing RL frameworks fall into two categories: co-located and disaggregated architectures. Co-located ones fail to address this resource tension by forcing both stages to share the same GPUs. Disaggregated architectures, without modifications of well-established RL algorithms, suffer from resource under-utilization. Meanwhile, preemptible GPU resources, i.e., spot instances on public clouds and spare capacity in production clusters, present significant cost-saving opportunities for accelerating RL workflows, if efficiently harvested for rollout. In this paper, we present RLBoost, a systematic solution for cost-efficient RL training that harvests preemptible GPU resources. Our key insight is that rollout's stateless and embarrassingly parallel nature aligns perfectly with preemptible and often fragmented resources. To efficiently utilize these resources despite frequent and unpredictable availability changes, RLBoost adopts a hybrid architecture with three key techniques: (1) adaptive rollout offload to dynamically adjust workloads on the reserved (on-demand) cluster, (2) pull-based weight transfer that quickly provisions newly available instances, and (3) token-level response collection and migration for efficient preemption handling and continuous load balancing. Extensive experiments show RLBoost increases training throughput by 1.51x-1.97x while improving cost efficiency by 28%-49% compared to using only on-demand GPU resources.

翻译：强化学习已成为解锁大型语言模型高级推理能力的关键技术。强化学习工作流包含交替进行的"推演"与"训练"阶段，二者具有本质不同的资源需求。推演阶段通常占据总执行时间的主导地位，但可通过多个独立实例实现高效扩展；相比之下，训练阶段需要具备全互联通信的紧耦合GPU集群。现有强化学习框架可分为两类：同址架构与解耦架构。同址架构强制两个阶段共享相同GPU资源，未能解决这种资源需求矛盾；而未经经典强化学习算法修改的解耦架构则存在资源利用率不足的问题。与此同时，抢占式GPU资源（如公有云的竞价实例与生产集群的闲置容量）若能有效用于推演阶段，将为加速强化学习工作流提供显著的成本节约机遇。本文提出RLBoost——一种通过利用抢占式GPU资源实现成本效益型强化训练的系统化解决方案。我们的核心洞见在于：推演阶段的无状态特性与高度并行化特征，恰好与抢占式且通常碎片化的资源特性高度契合。为应对资源可用性频繁且不可预测的变化，RLBoost采用混合架构并包含三项关键技术：(1) 自适应推演卸载机制，动态调整预留（按需）集群的工作负载；(2) 基于拉取的权重传输协议，快速配置新可用实例；(3) 令牌级响应收集与迁移机制，实现高效的抢占处理与持续负载均衡。大量实验表明，相较于仅使用按需GPU资源，RLBoost能将训练吞吐量提升1.51-1.97倍，同时将成本效益提高28%-49%。