Non-autoregressive approaches aim to improve the inference speed of translation models, particularly those that generate output in a one-pass forward manner. However, these approaches often suffer from a significant drop in translation quality compared to autoregressive models. This paper introduces a series of innovative techniques to enhance the translation quality of Non-Autoregressive Translation (NAT) models while maintaining a substantial acceleration in inference speed. We propose fine-tuning Pretrained Multilingual Language Models (PMLMs) with the CTC loss to train NAT models effectively. Furthermore, we adopt the MASK insertion scheme for up-sampling instead of token duplication, and we present an embedding distillation method to further enhance performance. In our experiments, our model outperforms the baseline autoregressive model (Transformer \textit{base}) on multiple datasets, including WMT'14 DE$\leftrightarrow$EN, WMT'16 RO$\leftrightarrow$EN, and IWSLT'14 DE$\leftrightarrow$EN. Notably, our model achieves better performance than the baseline autoregressive model on the IWSLT'14 En$\leftrightarrow$De and WMT'16 En$\leftrightarrow$Ro datasets, even without using distillation data during training. It is worth highlighting that on the IWSLT'14 DE$\rightarrow$EN dataset, our model achieves an impressive BLEU score of 39.59, setting a new state-of-the-art performance. Additionally, our model exhibits a remarkable speed improvement of 16.35 times compared to the autoregressive model.

翻译：非自回归方法旨在提高翻译模型的推理速度，尤其是那些以一次性前向方式生成输出的模型。然而，与非自回归方法相比，这些方法往往在翻译质量上出现显著下降。本文介绍了一系列创新技术，旨在提升非自回归翻译（NAT）模型的翻译质量，同时保持推理速度的大幅提升。我们提出使用连接时序分类（CTC）损失对预训练多语言语言模型（PMLMs）进行微调，以有效训练NAT模型。此外，我们采用MASK插入方案进行上采样，取代了传统的复制标记方法，并提出了一种嵌入蒸馏方法以进一步提升性能。在实验中，我们的模型在多个数据集上（包括WMT'14 DE↔EN、WMT'16 RO↔EN和IWSLT'14 DE↔EN）优于基线自回归模型（Transformer \textit{base}）。值得注意的是，即使在训练过程中未使用蒸馏数据，我们的模型在IWSLT'14 En↔De和WMT'16 En↔Ro数据集上也取得了优于基线自回归模型的性能。值得强调的是，在IWSLT'14 DE→EN数据集上，我们的模型达到了令人瞩目的39.59 BLEU分数，树立了新的最优性能标杆。此外，与自回归模型相比，我们的模型实现了16.35倍的惊人速度提升。