On-policy distillation (OPD) has become a core technique in the post-training of large language models, yet its training dynamics remain poorly understood. This paper provides a systematic investigation of OPD dynamics and mechanisms. We first identify that two conditions govern whether OPD succeeds or fails: (i) the student and teacher should share compatible thinking patterns; and (ii) even with consistent thinking patterns and higher scores, the teacher must offer genuinely new capabilities beyond what the student has seen during training. We validate these findings through weak-to-strong reverse distillation, showing that same-family 1.5B and 7B teachers are distributionally indistinguishable from the student's perspective. Probing into the token-level mechanism, we show that successful OPD is characterized by progressive alignment on high-probability tokens at student-visited states, a small shared token set that concentrates most of the probability mass (97%-99%). We further propose two practical strategies to recover failing OPD: off-policy cold start and teacher-aligned prompt selection. Finally, we show that OPD's apparent free lunch of dense token-level reward comes at a cost, raising the question of whether OPD can scale to long-horizon distillation.

翻译：同策略蒸馏已成为大型语言模型后训练中的核心技术，但其训练动态仍未被充分理解。本文系统研究了同策略蒸馏的动态过程与机制。我们首先识别出决定同策略蒸馏成败的两个条件：（i）学生模型与教师模型应具备兼容的思维模式；（ii）即使在思维模式一致且教师模型得分更高的情况下，教师模型也必须提供学生在训练中未曾接触过的真正新能力。我们通过弱到强的反向蒸馏验证了这些发现，表明同家族1.5B和7B教师模型在学生视角下具有分布不可区分性。在探索词元级机制时，我们证明成功的同策略蒸馏表现为学生访问状态下高概率词元的渐进对齐，其中一个小型共享词元集聚集了大部分概率质量（97%-99%）。我们进一步提出两种实用策略来恢复失效的同策略蒸馏：离策略冷启动和基于教师对齐的提示选择。最后，我们表明同策略蒸馏表面上的"免费午餐"——密集的词元级奖励——实则伴随着代价，这引发了同策略蒸馏能否扩展至长程蒸馏的疑问。