Recent advances in large language model (LLM) reasoning through reinforcement learning rely on annotated datasets for verifiable rewards, which may limit models' ability to surpass human-level performance. While self-play offers a promising alternative, existing approaches depend on external verifiers or cannot learn open-endedly. We present Open-Ended Self-Improving Reasoner (OpenSIR), a self-play framework where an LLM learns to generate and solve novel problems by alternating teacher and student roles without external supervision. To generate novel problems, OpenSIR optimises for both difficulty and diversity, rewarding problems that challenge appropriately while exploring distinct concepts, enabling open-ended mathematical discovery. Starting from a single trivial seed problem, OpenSIR substantially improves instruction models: Llama-3.2-3B-Instruct advances from 73.9 to 78.3 on GSM8K, and from 28.8 to 34.4 on College Math, while Gemma-2-2B-Instruct rises from 38.5 to 58.7 on GSM8K. Our analyses reveal that OpenSIR achieves open-ended learning through co-evolving teacher-student roles that adaptively calibrate difficulty and drive diverse exploration, progressing autonomously from basic to advanced mathematics.

翻译：近期，大型语言模型（LLM）通过强化学习在推理方面取得的进展依赖于带标注数据集来获取可验证的奖励，这可能限制模型超越人类水平性能的能力。尽管自我博弈提供了一种有前景的替代方案，但现有方法依赖于外部验证器或无法进行开放式学习。我们提出了开放式自我改进推理器（OpenSIR），这是一种自我博弈框架，其中LLM通过交替扮演教师和学生角色，在没有外部监督的情况下学习生成和解决新颖问题。为了生成新颖问题，OpenSIR针对难度和多样性进行优化，奖励那些在挑战性适度的同时探索不同概念的问题，从而实现开放式数学发现。从一个简单的种子问题开始，OpenSIR显著改进了指令模型：Llama-3.2-3B-Instruct在GSM8K上的得分从73.9提升至78.3，在College Math上从28.8提升至34.4；而Gemma-2-2B-Instruct在GSM8K上从38.5提升至58.7。我们的分析表明，OpenSIR通过协同演化的教师-学生角色实现开放式学习，这些角色自适应地校准难度并驱动多样化探索，从而自主地从基础数学进阶到高级数学。