Non-autoregressive (NAR) language models are known for their low latency in neural machine translation (NMT). However, a performance gap exists between NAR and autoregressive models due to the large decoding space and difficulty in capturing dependency between target words accurately. Compounding this, preparing appropriate training data for NAR models is a non-trivial task, often exacerbating exposure bias. To address these challenges, we apply reinforcement learning (RL) to Levenshtein Transformer, a representative edit-based NAR model, demonstrating that RL with self-generated data can enhance the performance of edit-based NAR models. We explore two RL approaches: stepwise reward maximization and episodic reward maximization. We discuss the respective pros and cons of these two approaches and empirically verify them. Moreover, we experimentally investigate the impact of temperature setting on performance, confirming the importance of proper temperature setting for NAR models' training.

翻译：非自回归（NAR）语言模型因其在神经机器翻译（NMT）中的低延迟特性而闻名。然而，由于解码空间庞大且难以准确捕捉目标词之间的依赖关系，NAR模型与自回归模型之间存在性能差距。更棘手的是，为NAR模型准备合适的训练数据是一项具有挑战性的任务，往往加剧曝光偏差。为应对这些问题，我们将强化学习（RL）应用于Levenshtein Transformer——一种代表性的基于编辑的NAR模型，证明使用自生成数据的RL能够提升基于编辑的NAR模型的性能。我们探索了两种强化学习方法：逐步奖励最大化和片段奖励最大化。我们讨论了这两种方法的各自优缺点，并进行了实证验证。此外，我们通过实验研究了温度设置对性能的影响，确认了适当的温度设置对NAR模型训练的重要性。