Automatic evaluation of machine translation (MT) is a critical tool driving the rapid iterative development of MT systems. While considerable progress has been made on estimating a single scalar quality score, current metrics lack the informativeness of more detailed schemes that annotate individual errors, such as Multidimensional Quality Metrics (MQM). In this paper, we help fill this gap by proposing AutoMQM, a prompting technique which leverages the reasoning and in-context learning capabilities of large language models (LLMs) and asks them to identify and categorize errors in translations. We start by evaluating recent LLMs, such as PaLM and PaLM-2, through simple score prediction prompting, and we study the impact of labeled data through in-context learning and finetuning. We then evaluate AutoMQM with PaLM-2 models, and we find that it improves performance compared to just prompting for scores (with particularly large gains for larger models) while providing interpretability through error spans that align with human annotations.

翻译：机器翻译（MT）的自动评估是推动MT系统快速迭代发展的关键工具。尽管在估计单一标量质量分数方面取得了显著进展，但现有指标缺乏多维度质量指标（MQM）等更详细标注单个错误方案的丰富信息。本文通过提出AutoMQM方法填补了这一空白——该提示技术利用大型语言模型（LLM）的推理与上下文学习能力，引导其识别并分类翻译中的错误。我们首先评估了PaLM、PaLM-2等最新LLM在简单分数预测提示下的表现，并研究了通过上下文学习与微调利用标注数据的影响。随后，我们基于PaLM-2模型评估AutoMQM，发现相较于仅提示生成分数（尤其对更大模型而言），该方法在性能提升的同时，还能通过对齐人工标注的错误片段提供可解释性。