Being one of the IR-NAT (Iterative-refinemennt-based NAT) frameworks, the Conditional Masked Language Model (CMLM) adopts the mask-predict paradigm to re-predict the masked low-confidence tokens. However, CMLM suffers from the data distribution discrepancy between training and inference, where the observed tokens are generated differently in the two cases. In this paper, we address this problem with the training approaches of error exposure and consistency regularization (EECR). We construct the mixed sequences based on model prediction during training, and propose to optimize over the masked tokens under imperfect observation conditions. We also design a consistency learning method to constrain the data distribution for the masked tokens under different observing situations to narrow down the gap between training and inference. The experiments on five translation benchmarks obtains an average improvement of 0.68 and 0.40 BLEU scores compared to the base models, respectively, and our CMLMC-EECR achieves the best performance with a comparable translation quality with the Transformer. The experiments results demonstrate the effectiveness of our method.

翻译：属于迭代精炼型非自回归翻译（IR-NAT）框架的条件掩码语言模型（CMLM）采用"掩码-预测"范式，对掩码的低置信度词元进行重新预测。然而，CMLM在训练与推理阶段面临数据分布差异问题，即观测到的词元在两种场景下的生成方式不同。本文通过错误暴露与一致性正则化（EECR）的训练方法解决该问题。我们基于模型预测构建混合序列，并提出在不完全观测条件下对掩码词元进行优化。同时设计一致性学习方法，约束不同观测情境下掩码词元的数据分布，以缩小训练与推理之间的差距。在五个翻译基准上的实验表明，与基础模型相比，分别取得了0.68和0.40 BLEU值的平均提升，我们的CMLMC-EECR在达到与Transformer相当翻译质量的前提下实现了最优性能。实验结果证明了该方法的有效性。