Recently, very large pre-trained models achieve state-of-the-art results in various natural language processing (NLP) tasks, but their size makes it more challenging to apply them in resource-constrained environments. Compression techniques allow to drastically reduce the size of the models and therefore their inference time with negligible impact on top-tier metrics. However, the general performance averaged across multiple tasks and/or languages may hide a drastic performance drop on under-represented features, which could result in the amplification of biases encoded by the models. In this work, we assess the impact of compression methods on Multilingual Neural Machine Translation models (MNMT) for various language groups, gender, and semantic biases by extensive analysis of compressed models on different machine translation benchmarks, i.e. FLORES-101, MT-Gender, and DiBiMT. We show that the performance of under-represented languages drops significantly, while the average BLEU metric only slightly decreases. Interestingly, the removal of noisy memorization with compression leads to a significant improvement for some medium-resource languages. Finally, we demonstrate that compression amplifies intrinsic gender and semantic biases, even in high-resource languages. Code: https://github.com/alirezamshi/bias-compressedMT

翻译：近年来，超大型预训练模型在各种自然语言处理（NLP）任务中取得了最先进的成果，但其规模使得在资源受限环境中应用这些模型更加困难。压缩技术能够大幅减小模型规模，从而减少推理时间，同时对高级指标的影响微乎其微。然而，跨多个任务和/或语言的平均整体性能可能掩盖了在代表性不足的特征上的显著性能下降，这可能导致模型编码的偏见被放大。在本工作中，我们通过在不同机器翻译基准（即FLORES-101、MT-Gender和DiBiMT）上对压缩模型进行广泛分析，评估了压缩方法对多语言神经机器翻译模型（MNMT）在各类语言组、性别和语义偏见方面的影响。我们表明，代表性不足的语言的性能显著下降，而平均BLEU指标仅略有降低。有趣的是，通过压缩去除嘈杂的记忆导致一些中等资源语言的性能显著提升。最后，我们证明压缩放大了内在的性别和语义偏见，即使在高等资源语言中也是如此。代码：https://github.com/alirezamshi/bias-compressedMT