Standard on-policy distillation (OPD) for large language models estimates the reverse-KL objective using student-sampled tokens, yielding an unbiased single-sample Monte Carlo estimator that avoids vocabulary-wide computation. However, we show that this estimator suffers from severe training pathologies in practice: sample inefficiency, unstable generation dynamics, and a substantial performance gap compared to exact full-vocabulary OPD. Reward-level diagnosis traces these pathologies to the log-ratio reward, which is unbounded by construction, producing extremely high-variance gradients concentrated at early positions and persisting throughout training; standard post-hoc scaling fail as they operate only after this distortion occurs. To solve this problem, we propose PowerOPD: a family of natively bounded, sign-consistent rewards from the Box-Cox power transformation, parameterized by alpha > 0, of which the log-ratio is the degenerate alpha -> 0 limit. Across six mathematical reasoning benchmarks and four Qwen3 teacher-student pairs, PowerOPD achieves benchmark-averaged Avg@8/Pass@8 gains of up to +6.37/+5.71 over vanilla OPD, +3.01/+3.54 over post-hoc stabilization, and +2.59/+8.90 over full-vocabulary OPD, while reducing wall-clock time by 59.2% and peak GPU memory by 23.1%. Larger alpha generally improves accuracy, consistently shortens responses, and keeps gradient norms more than 3,000x smaller than vanilla OPD.

翻译：标准大语言模型同策略蒸馏（OPD）通过学生采样词元估计反向KL散度目标，从而获得避免全词表计算的单样本无偏蒙特卡洛估计量。然而，我们证明该估计量在实践中存在严重训练病理现象：样本效率低下、生成动态不稳定，以及相较于精确全词表OPD存在显著性能差距。奖励层面诊断将这些病理归因于对数比值奖励——该奖励在构造上无界，会产生集中在早期位置且贯穿整个训练过程的高方差梯度；标准后处理缩放方法仅在畸变发生后生效，难以奏效。为解决此问题，我们提出PowerOPD：基于Box-Cox幂变换的原始有界、符号一致奖励族，由参数α>0控制，其中对数比值是其退化极限α→0。在六个数学推理基准和四组Qwen3师生模型对上的实验表明，PowerOPD在基准平均Avg@8/Pass@8指标上较原始OPD提升最高达+6.37/+5.71，较后处理稳定方法提升+3.01/+3.54，较全词表OPD提升+2.59/+8.90，同时将运行时间缩短59.2%，峰值GPU内存降低23.1%。较大α值通常能提升准确率、持续缩短响应长度，并使梯度范数比原始OPD小3000倍以上。