On-policy knowledge distillation (OPD) trains a student on its own rollouts under token-level supervision from a teacher. Not all token positions matter equally, but existing views of token importance are incomplete. We ask a direct question: which tokens carry the most useful learning signal in OPD? Our answer is that informative tokens come from two regions: positions with high student entropy, and positions with low student entropy plus high teacher--student divergence, where the student is overconfident and wrong. Empirically, student entropy is a strong first-order proxy: retaining $50\%$ of tokens with entropy-based sampling matches or exceeds all-token training while reducing peak memory by up to $47\%$. But entropy alone misses a second important region. When we isolate low-entropy, high-divergence tokens, training on fewer than $10\%$ of all tokens nearly matches full-token baselines, showing that overconfident tokens carry dense corrective signal despite being nearly invisible to entropy-only rules. We organize these findings with TIP (Token Importance in on-Policy distillation), a two-axis taxonomy over student entropy and teacher--student divergence, and give a theoretical explanation for why entropy is useful yet structurally incomplete. This view motivates type-aware token selection rules that combine uncertainty and disagreement. We validate this picture across three teacher--student pairs spanning Qwen3, Llama, and Qwen2.5 on MATH-500 and AIME 2024/2025, and on the DeepPlanning benchmark for long-horizon agentic planning, where Q3-only training on $<$$20\%$ of tokens surpasses full-token OPD. Our experiments are implemented by extending the OPD repository https://github.com/HJSang/OPSD_OnPolicyDistillation, which supports memory-efficient distillation of larger models under limited GPU budgets.

翻译：在线策略知识蒸馏（OPD）在教师模型的令牌级监督下，基于学生模型自身的轨迹对其进行训练。并非所有令牌位置的重要性相同，但现有关于令牌重要性的观点并不完整。我们提出一个直接问题：在OPD中，哪些令牌携带最有用的学习信号？答案是，信息性令牌来自两个区域：具有高学生熵的位置，以及具有低学生熵但高师生分歧的位置（即学生过度自信且错误的情况）。实验表明，学生熵是一个有效的首要代理指标：基于熵采样保留50%的令牌，其效果可与全令牌训练相当或更优，同时将峰值内存降低高达47%。但仅凭熵会遗漏第二个关键区域。当我们分离出低熵、高分歧令牌时，基于不到10%的令牌进行训练几乎能匹配全令牌基线，表明过度自信的令牌携带密集的修正信号，尽管它们在仅依赖熵的规则中几乎不可见。我们将这些发现组织成TIP（基于策略蒸馏中的令牌重要性），这是一个基于学生熵与师生分歧的双轴分类框架，并从理论上解释了熵为何有效但结构上不完整。这一观点启发了结合不确定性与分歧的类型感知令牌选择规则。我们在Qwen3、Llama和Qwen2.5的三种师生模型组合上，在MATH-500和AIME 2024/2025数据集，以及面向长时域智能体规划的DeepPlanning基准上验证了这一框架。在DeepPlanning中，仅基于不到20%令牌的Q3训练即超越了全令牌OPD。我们的实验通过扩展OPD仓库https://github.com/HJSang/OPSD_OnPolicyDistillation实现，该仓库支持在有限GPU预算下对更大模型进行内存高效的蒸馏。