Iterative self-improvement fine-tunes an autoregressive large language model (LLM) on reward-verified outputs generated by the LLM itself. In contrast to the empirical success of self-improvement, the theoretical foundation of this generative, iterative procedure in a practical, finite-sample setting remains limited. We make progress toward this goal by modeling each round of self-improvement as maximum-likelihood fine-tuning on a reward-filtered distribution and deriving finite-sample guarantees for the expected reward. Our analysis reveals an explicit feedback loop where better models accept more data per iteration, supporting sustained self-improvement while explaining eventual saturation of such improvement. Adopting a task-centric view by considering reasoning tasks with multiple difficulty levels, we further prove quantifiable conditions on model initialization, task difficulty, and sample budget where easy-to-hard curricula provably achieve better guarantees than training on fixed mixtures of tasks. Our analyses are validated via Monte-Carlo simulations and controlled experiments on graph-based reasoning tasks.

翻译：迭代式自我改进通过在由自回归大语言模型（LLM）生成且经过奖励验证的输出上对模型进行微调。尽管自我改进在实践中取得了显著成效，但针对这种生成式迭代过程在有限样本实际场景中的理论基础仍较为薄弱。本研究通过将每一轮自我改进建模为对奖励筛选分布的极大似然微调，并推导期望奖励的有限样本保证，在此方向取得了进展。分析揭示了一个显式反馈循环：性能更优的模型在每次迭代中能够接受更多数据，这既支撑了持续的自我改进，也解释了此类改进最终趋于饱和的现象。通过采用任务中心化视角——考虑具有多难度层次的推理任务，我们进一步证明了在模型初始化、任务难度及样本预算满足特定量化条件时，易到难课程学习策略能够获得比固定任务混合训练更优的理论保证。所有分析均通过蒙特卡洛模拟及基于图结构推理任务的受控实验得到验证。