Large language models have led to significant progress across many NLP tasks, although their massive sizes often incur substantial computational costs. Distillation has become a common practice to compress these large and highly capable models into smaller, more efficient ones. Many existing language model distillation methods can be viewed as behavior cloning from the perspective of imitation learning or inverse reinforcement learning. This viewpoint has inspired subsequent studies that leverage (inverse) reinforcement learning techniques, including variations of behavior cloning and temporal difference learning methods. Rather than proposing yet another specific temporal difference method, we introduce a general framework for temporal difference-based distillation by exploiting the distributional sparsity of the teacher model. Specifically, it is often observed that language models assign most probability mass to a small subset of tokens. Motivated by this observation, we design a temporal difference learning framework that operates on a reduced action space (a subset of vocabulary), and demonstrate how practical algorithms can be derived and the resulting performance improvements.

翻译：大型语言模型已在众多自然语言处理任务中取得显著进展，但其庞大的参数量通常伴随着可观的计算开销。蒸馏已成为将这些大规模高性能模型压缩为更小型高效模型的常用技术。现有许多语言模型蒸馏方法可从模仿学习或逆强化学习的视角视为行为克隆。这一观点启发了后续研究利用（逆）强化学习技术，包括行为克隆的变体与时序差分学习方法。不同于提出另一种具体的时序差分方法，本文通过利用教师模型的分布稀疏性，提出了基于时序差分的蒸馏通用框架。具体而言，语言模型通常将大部分概率质量分配给少量词汇子集的现象已被广泛观测。受此启发，我们设计了一种在约简动作空间（词汇子集）上操作的时序差分学习框架，并论证了如何推导出实用算法及其带来的性能提升。