Knowledge distillation (KD) is a well-known method for compressing neural models. However, works focusing on distilling knowledge from large multilingual neural machine translation (MNMT) models into smaller ones are practically nonexistent, despite the popularity and superiority of MNMT. This paper bridges this gap by presenting an empirical investigation of knowledge distillation for compressing MNMT models. We take Indic to English translation as a case study and demonstrate that commonly used language-agnostic and language-aware KD approaches yield models that are 4-5x smaller but also suffer from performance drops of up to 3.5 BLEU. To mitigate this, we then experiment with design considerations such as shallower versus deeper models, heavy parameter sharing, multi-stage training, and adapters. We observe that deeper compact models tend to be as good as shallower non-compact ones, and that fine-tuning a distilled model on a High-Quality subset slightly boosts translation quality. Overall, we conclude that compressing MNMT models via KD is challenging, indicating immense scope for further research.

翻译：知识蒸馏（KD）是一种广为人知的神经模型压缩方法。然而，尽管多语言神经机器翻译（MNMT）模型具有广泛适用性和优越性能，目前几乎不存在将知识从大型MNMT模型蒸馏至小型模型的相关研究。本文通过系统性地实证研究知识蒸馏在MNMT模型压缩中的应用，填补了上述空白。我们以印度语到英语的翻译为案例，证明常用的语言无关与语言感知知识蒸馏方法虽能实现4-5倍模型压缩，但会导致BLEU值最高下降3.5个点。为缓解这一问题，我们进一步实验了浅层与深层模型、强参数共享、多阶段训练及适配器等多种设计策略。研究结果表明，深层紧凑模型的表现可媲美浅层非紧凑模型，且在高质量子集上微调蒸馏模型能小幅提升翻译质量。综上所述，通过知识蒸馏压缩MNMT模型具有显著挑战性，凸显了该方向未来研究的广阔空间。