Reinforcement learning with verifiable rewards (RLVR) has become a central component of large language model (LLM) post-training. Unlike supervised fine-tuning (SFT), RLVR lets an LLM generate multiple candidate solutions and reinforces those that lead to a verifiably correct final answer. However, in practice, RLVR often requires thousands of training steps to reach strong performance, incurring substantial computation largely attributed to prolonged exploration. In this work, we make a surprising observation: during RLVR, LLMs evolve in a strongly linear manner. Specifically, both model weights and model output log-probabilities exhibit strong linear correlations with RL training steps. This suggests that RLVR predominantly amplifies trends that emerge early in training, rather than continuously discovering new behaviors throughout the entire optimization trajectory. Motivated by this linearity, we investigate whether future model states can be predicted from intermediate checkpoints via extrapolation, avoiding continued expensive training. We show that Weight Extrapolation produces models with performance comparable to standard RL training while requiring significantly less computation. Moreover, Logits Extrapolation consistently outperforms continued RL training on mathematics and code benchmarks by extrapolating beyond the step range where RL training remains stable. Our code is available at https://github.com/Miaow-Lab/RLVR-Linearity

翻译：可验证奖励强化学习已成为大语言模型后训练的核心组成部分。与监督微调不同，RLVR允许大语言模型生成多个候选解决方案，并对那些能产生可验证正确答案的解决方案进行强化。然而在实践中，RLVR通常需要数千个训练步骤才能达到强劲性能，这主要归因于漫长的探索过程所产生的巨大计算开销。本研究发现了一个令人惊讶的现象：在RLVR训练过程中，大语言模型呈现出强烈的线性演化特征。具体而言，模型权重和模型输出的对数概率均与RL训练步数表现出强烈的线性相关性。这表明RLVR主要放大的是训练早期出现的趋势，而非在整个优化轨迹中持续发现新的行为模式。受此线性特性的启发，我们探究是否可以通过中间检查点的外推来预测未来模型状态，从而避免持续的高成本训练。实验证明，权重外推法生成的模型性能与标准RL训练相当，同时显著降低了计算需求。此外，在数学和代码基准测试中，通过对超出RL训练稳定区间的步数范围进行外推，对数概率外推法持续优于持续RL训练。代码已开源：https://github.com/Miaow-Lab/RLVR-Linearity