The advent of scalable deep models and large datasets has improved the performance of Neural Machine Translation. Knowledge Distillation (KD) enhances efficiency by transferring knowledge from a teacher model to a more compact student model. However, KD approaches to Transformer architecture often rely on heuristics, particularly when deciding which teacher layers to distill from. In this paper, we introduce the 'Align-to-Distill' (A2D) strategy, designed to address the feature mapping problem by adaptively aligning student attention heads with their teacher counterparts during training. The Attention Alignment Module in A2D performs a dense head-by-head comparison between student and teacher attention heads across layers, turning the combinatorial mapping heuristics into a learning problem. Our experiments show the efficacy of A2D, demonstrating gains of up to +3.61 and +0.63 BLEU points for WMT-2022 De->Dsb and WMT-2014 En->De, respectively, compared to Transformer baselines.

翻译：可扩展深度模型和大规模数据集的涌现提升了神经机器翻译的性能。知识蒸馏通过将知识从教师模型迁移到更紧凑的学生模型来提高效率。然而，针对Transformer架构的知识蒸馏方法常依赖启发式策略，尤其是在决定从哪些教师层进行蒸馏时。本文提出“对齐到蒸馏”（A2D）策略，旨在通过训练过程中自适应地对齐学生与教师的注意力头来解决特征映射问题。A2D中的注意力对齐模块对学生和教师模型跨层的注意力头执行密集的逐头比较，将组合式映射启发转化为学习问题。实验表明，A2D策略的有效性体现在：与Transformer基线模型相比，在WMT-2022 De→Dsb和WMT-2014 En→De任务上分别获得了高达+3.61和+0.63 BLEU值的提升。