Self-distillation (SD) offers a promising path for adapting large language models (LLMs) without relying on stronger external teachers. However, SD in autoregressive LLMs remains challenging because self-generated trajectories are free-form, correctness is task-dependent, and plausible rationales can still provide unstable or unreliable supervision. Existing methods mainly examine isolated design choices, leaving their effectiveness, roles, and interactions unclear. In this paper, we propose UniSD, a unified framework to systematically study self-distillation. UniSD integrates complementary mechanisms that address supervision reliability, representation alignment, and training stability, including multi-teacher agreement, EMA teacher stabilization, token-level contrastive learning, feature matching, and divergence clipping. Across six benchmarks and six models from three model families, UniSD reveals when self-distillation improves over static imitation, which components drive the gains, and how these components interact across tasks. Guided by these insights, we construct UniSDfull, an integrated pipeline that combines complementary components and achieves the strongest overall performance, improving over the base model by +5.4 points and the strongest baseline by +2.8 points. Extensive evaluation highlights self-distillation as a practical and steerable approach for efficient LLM adaptation without stronger external teachers.

翻译：自蒸馏（SD）为大型语言模型（LLM）的适配提供了一条无需依赖更强外部教师模型的前景路径。然而，在自回归LLM中，由于自生成轨迹的自由形式特性、正确性对任务的依赖性以及合理解释仍可能提供不稳定或不可靠监督等问题，自蒸馏仍面临挑战。现有方法主要考察孤立的設計选择，使得它们的有效性、作用及交互关系尚不明确。本文提出UniSD，一个用于系统研究自蒸馏的统一框架。UniSD整合了多种互补机制，包括多教师一致性、EMA教师稳定化、标记级对比学习、特征匹配及散度裁剪，从而解决监督可靠性、表示对齐与训练稳定性问题。在三个模型家族的六个基准测试与六个模型上的实验表明，UniSD揭示了自蒸馏何时优于静态模仿、哪些组件驱动性能提升以及这些组件如何在任务间相互作用。基于这些洞见，我们构建了UniSDfull——一个整合互补组件的集成管道，实现了整体最优性能，较基础模型提升+5.4分，较最强基线提升+2.8分。广泛评估凸显了自蒸馏作为无需更强外部教师模型即可高效适配LLM的实用且可控方法的潜力。