Almost all frameworks for the manual or automatic evaluation of machine translation characterize the quality of an MT output with a single number. An exception is the Multidimensional Quality Metrics (MQM) framework which offers a fine-grained ontology of quality dimensions for scoring (such as style, fluency, accuracy, and terminology). Previous studies have demonstrated the feasibility of MQM annotation but there are, to our knowledge, no computational models that predict MQM scores for novel texts, due to a lack of resources. In this paper, we address these shortcomings by (a) providing a 1200-sentence MQM evaluation benchmark for the language pair English-Korean and (b) reframing MT evaluation as the multi-task problem of simultaneously predicting several MQM scores using SOTA language models, both in a reference-based MT evaluation setup and a reference-free quality estimation (QE) setup. We find that reference-free setup outperforms its counterpart in the style dimension while reference-based models retain an edge regarding accuracy. Overall, RemBERT emerges as the most promising model. Through our evaluation, we offer an insight into the translation quality in a more fine-grained, interpretable manner.

翻译：几乎所有用于人工或自动评估机器翻译的框架，都以单一数值表征翻译输出质量。多维质量指标（MQM）框架是例外，它提供了细粒度的质量维度评分本体（如风格、流畅度、准确性和术语）。先前研究已证明MQM标注的可行性，但据我们所知，由于缺乏资源，目前尚未存在能预测新文本MQM分数的计算模型。本文通过以下方式弥补上述不足：(a) 为英韩语言对提供包含1200句的MQM评估基准测试集；(b) 将机器翻译评估重构为多任务问题，利用最新语言模型同时预测多个MQM分数，涵盖基于参考的翻译评估和无参考质量估计两种设置。研究发现：无参考设置在风格维度上优于有参考设置，而基于参考的模型在准确性方面仍具优势。总体而言，RemBERT展现出最佳潜力。通过本评估，我们以更细粒度、可解释的方式深入揭示了翻译质量特征。