Recent research in decoding methods for Natural Language Generation (NLG) tasks has shown that MAP decoding is not optimal, because model probabilities do not always align with human preferences. Stronger decoding methods, including Quality Estimation (QE) reranking and Minimum Bayes' Risk (MBR) decoding, have since been proposed to mitigate the model-perplexity-vs-quality mismatch. While these decoding methods achieve state-of-the-art performance, they are prohibitively expensive to compute. In this work, we propose MBR finetuning and QE finetuning which distill the quality gains from these decoding methods at training time, while using an efficient decoding algorithm at inference time. Using the canonical NLG task of Neural Machine Translation (NMT), we show that even with self-training, these finetuning methods significantly outperform the base model. Moreover, when using an external LLM as a teacher model, these finetuning methods outperform finetuning on human-generated references. These findings suggest new ways to leverage monolingual data to achieve improvements in model quality that are on par with, or even exceed, improvements from human-curated data, while maintaining maximum efficiency during decoding.

翻译：自然语言生成任务中解码方法的最新研究表明，最大后验概率解码并非最优选择，因为模型概率并不总是与人类偏好一致。此后，包括质量估计重排序和最小贝叶斯风险解码在内的更强解码方法被提出，以缓解模型困惑度与生成质量不匹配的问题。尽管这些解码方法实现了最先进的性能，但其计算成本过高。在这项工作中，我们提出了MBR微调和QE微调方法，在训练时将来自这些解码方法的质量收益进行蒸馏，同时在推理时采用高效解码算法。以神经机器翻译这一经典NLG任务为例，我们证明即使通过自训练，这些微调方法的性能也显著优于基础模型。此外，当使用外部大语言模型作为教师模型时，这些微调方法的表现优于使用人工生成参考进行微调的效果。这些发现提出了一种利用单语数据的新途径，能够在解码过程中保持最高效率的同时，实现与人工精选数据相当甚至更优的模型质量提升。