Reinforcement learning (RL) offers a principled way to enhance the reasoning capabilities of large language models, yet its effectiveness hinges on training signals that remain informative as models evolve. In practice, RL progress often slows when task difficulty becomes poorly aligned with model capability, or when training is dominated by a narrow set of recurring problem patterns. To jointly address these issues, we propose SCALER (Synthetic sCalable Adaptive Learning Environment for Reasoning), a framework that sustains effective learning signals through adaptive environment design. SCALER introduces a scalable synthesis pipeline that converts real-world programming problems into verifiable reasoning environments with controllable difficulty and unbounded instance generation, enabling RL training beyond finite datasets while preserving strong correctness guarantees. Building on this, SCALER further employs an adaptive multi-environment RL strategy that dynamically adjusts instance difficulty and curates the active set of environments to track the model's capability frontier and maintain distributional diversity. This co-adaptation prevents reward sparsity, mitigates overfitting to narrow task patterns, and supports sustained improvement throughout training. Extensive experiments show that SCALER consistently outperforms dataset-based RL baselines across diverse reasoning benchmarks and exhibits more stable, long-horizon training dynamics.

翻译：强化学习为增强大语言模型的推理能力提供了一种原则性方法，但其有效性取决于训练信号能否随着模型演化保持信息量。实践中，当任务难度与模型能力严重不匹配，或训练被少量重复出现的问题模式主导时，强化学习的进展往往会放缓。为协同解决这些问题，我们提出了SCALER（用于推理的合成可扩展自适应学习环境），这是一个通过自适应环境设计来维持有效学习信号的框架。SCALER引入了一个可扩展的合成流程，将现实世界的编程问题转化为具有可控难度和无限实例生成能力的可验证推理环境，从而使得强化学习训练能够超越有限数据集，同时保持强有力的正确性保证。在此基础上，SCALER进一步采用了一种自适应多环境强化学习策略，该策略动态调整实例难度并精选活跃环境集合，以追踪模型的能力边界并保持分布多样性。这种协同适应机制防止了奖励稀疏性，减轻了对狭窄任务模式的过拟合，并支持在整个训练过程中持续改进。大量实验表明，SCALER在多种推理基准测试中始终优于基于数据集的强化学习基线，并展现出更稳定、更长视野的训练动态。