The standard post-training recipe for large reasoning models, supervised fine-tuning followed by reinforcement learning (SFT-then-RL), may limit the benefits of the RL stage: while SFT imitates expert demonstrations, it often causes overconfidence and reduces generation diversity, leaving RL with a narrowed solution space to explore. Adding entropy regularization during SFT is not a cure-all; it tends to flatten token distributions toward uniformity, increasing entropy without improving meaningful exploration capability. In this paper, we propose CurioSFT, an entropy-preserving SFT method designed to enhance exploration capabilities through intrinsic curiosity. It consists of (a) Self-Exploratory Distillation, which distills the model toward a self-generated, temperature-scaled teacher to encourage exploration within its capability; and (b) Entropy-Guided Temperature Selection, which adaptively adjusts distillation strength to mitigate knowledge forgetting by amplifying exploration at reasoning tokens while stabilizing factual tokens. Extensive experiments on mathematical reasoning tasks demonstrate that, in SFT stage, CurioSFT outperforms the vanilla SFT by 2.5 points on in-distribution tasks and 2.9 points on out-of-distribution tasks. We also verify that exploration capabilities preserved during SFT successfully translate into concrete gains in RL stage, yielding an average improvement of 5.0 points.

翻译：大型推理模型的标准后训练流程——监督微调后接强化学习（SFT-then-RL）——可能限制了强化学习阶段的收益：监督微调虽然模仿专家示范，但常导致模型过度自信并降低生成多样性，使得强化学习只能在一个狭窄的解空间中进行探索。在监督微调期间添加熵正则化并非万能良药；它倾向于使词元分布趋于均匀，虽增加了熵却未能提升有意义的探索能力。本文提出CurioSFT，一种旨在通过内在好奇心增强探索能力的熵保持监督微调方法。该方法包含两个核心组件：（a）自探索蒸馏，通过将模型向一个自生成的、经过温度缩放的教师模型进行蒸馏，以鼓励在其能力范围内的探索；（b）熵引导的温度选择，自适应地调整蒸馏强度，通过放大推理词元处的探索并稳定事实词元，以缓解知识遗忘。在数学推理任务上的大量实验表明，在监督微调阶段，CurioSFT在分布内任务上比原始监督微调高出2.5个百分点，在分布外任务上高出2.9个百分点。我们还验证了监督微调阶段所保留的探索能力能成功转化为强化学习阶段的具体收益，带来平均5.0个百分点的提升。