Reinforcement learning with verifiable rewards (RLVR) has recently unlocked strong reasoning capabilities in large language models (LLMs), triggering rapid exploration of new algorithms and data. However, RLVR training is notoriously inefficient: long-tailed rollouts, tool-induced stalls, and asymmetric resource requirements between rollout and training introduce substantial idle time that cannot be eliminated by job-local optimizations such as synchronous pipelining, asynchronous rollout, or colocated execution. We argue that this inefficiency is structural. While idle gaps are unavoidable within individual RLVR jobs, they are largely anti-correlated across jobs and therefore exploitable at the cluster level. Leveraging this observation, we present PlexRL, a cluster-level runtime for multiplexing unified LLM services across RLVR jobs. By centrally managing model placement, state transitions, and function-level scheduling under strict affinity constraints, PlexRL time-slices LLM execution across jobs to fill otherwise idle periods without expensive model migration. Our implementation and evaluations demonstrate that PlexRL significantly improves effective cluster capacity and reduces user GPU hour cost by maximum 37.58% while preserving algorithmic flexibility and introducing minimal per-job overhead.

翻译：基于可验证奖励的强化学习（RLVR）近期已解锁大型语言模型（LLMs）的强大推理能力，促使新算法与新数据的快速探索。然而，RLVR训练存在众所周知的低效问题：长尾采样、工具引发的阻塞以及采样与训练间不对称的资源需求，导致了大量空闲时间，这些时间无法通过作业局部优化（如同步流水线、异步采样或共置执行）消除。我们认为这种低效是结构性的。虽然空闲间隙在单个RLVR作业内部不可避免，但它们在不同作业间大体呈负相关，因此可在集群层面加以利用。基于这一观察，我们提出PlexRL，一种跨RLVR作业复用统一LLM服务的集群级运行时。通过在严格亲和性约束下集中管理模型放置、状态转换与函数级调度，PlexRL跨作业对LLM执行进行时间片切片，从而填充原本空闲的时段，且无需昂贵的模型迁移。我们的实现与评估表明，PlexRL在保持算法灵活性并引入极小单作业开销的同时，显著提升了有效集群容量，最大可减少用户GPU小时成本37.58%。