Reinforcement Learning (RL) has emerged as a critical technique for advancing modern Large Language Models (LLMs), yet existing synchronous RL systems face severe performance bottlenecks. The rollout phase, which dominates end-to-end iteration time, suffers from substantial long-tail latency and poor resource utilization due to inherent workload imbalance. We present Seer, a novel context learning RL system that addresses these challenges through a key observation: requests sharing the same prompt exhibit strong similarities in output lengths and response patterns. Leveraging this insight, Seer introduces three coordinated techniques: (1) divided rollout for dynamic load balancing, (2) context-aware scheduling to mitigate long-tail request delays, and (3) adaptive grouped speculative decoding to accelerate generation. These mechanisms work in concert to markedly reduce long-tail latency and improve resource efficiency during rollout. Evaluations on production-grade RL workloads demonstrate that Seer achieves up to 2.04$\times$ end-to-end rollout throughput improvement compared to the state-of-the-art synchronous RL systems, while notably reducing long-tail latency by 72-94%.

翻译：强化学习（RL）已成为推进现代大语言模型（LLM）发展的关键技术，然而现有同步RL系统面临严重的性能瓶颈。其中占据端到端迭代时间主导地位的推出阶段，由于固有的工作负载不均衡，导致严重的尾延迟和资源利用率低下问题。我们提出Seer，一种新型的上下文学习RL系统，通过关键观察解决上述挑战：共享相同提示词的请求在输出长度和响应模式上呈现强相似性。基于这一发现，Seer引入了三项协同技术：（1）分割式推出实现动态负载均衡，（2）上下文感知调度缓解长尾请求延迟，（3）自适应分组推测解码加速生成。这些机制协同运作，显著降低推出阶段的尾延迟并提升资源效率。在实际生产级RL工作负载上的评估表明，与现有最优同步RL系统相比，Seer可实现高达2.04倍的端到端推出吞吐量提升，同时将尾延迟显著降低72%-94%。