On-policy distillation (OPD) has become a core technique in the post-training of large language models, yet its training dynamics remain poorly understood. This paper provides a systematic investigation of OPD dynamics and mechanisms. We first identify that two conditions govern whether OPD succeeds or fails: (i) the student and teacher should share compatible thinking patterns; and (ii) even with consistent thinking patterns and higher scores, the teacher must offer genuinely new capabilities beyond what the student has seen during training. We validate these findings through weak-to-strong reverse distillation, showing that same-family 1.5B and 7B teachers are distributionally indistinguishable from the student's perspective. Probing into the token-level mechanism, we show that successful OPD is characterized by progressive alignment on high-probability tokens at student-visited states, a small shared token set that concentrates most of the probability mass (97%-99%). We further propose two practical strategies to recover failing OPD: off-policy cold start and teacher-aligned prompt selection. Finally, we show that OPD's apparent free lunch of dense token-level reward comes at a cost, raising the question of whether OPD can scale to long-horizon distillation.

翻译：有策略蒸馏已成为大型语言模型后训练中的核心技术，但其训练动力学仍未被充分理解。本文系统研究了有策略蒸馏的动力学与机制。我们首先识别出决定有策略蒸馏成功与否的两个条件：（i）学生模型与教师模型应共享兼容的思维模式；（ii）即使思维模式一致且得分更高，教师模型也必须提供学生模型在训练中未曾见过的全新能力。我们通过弱到强反向蒸馏验证了这些发现，证明同族1.5B与7B教师模型从学生模型视角看具有分布不可区分性。通过探究词元级机制，我们证明成功的有策略蒸馏的特征是：在学生模型访问的状态下，高概率词元逐步对齐，且一个较小的共享词元集集中了大部分概率质量（97%-99%）。我们进一步提出两种恢复失败有策略蒸馏的实用策略：非策略冷启动与教师对齐提示选择。最后，我们证明有策略蒸馏看似免费的密集词元级奖励午餐实则是有代价的，这引发了有策略蒸馏能否扩展至长程蒸馏的疑问。