Quality Estimation (QE) is the task of predicting the quality of Machine Translation (MT) system output, without using any gold-standard translation references. State-of-the-art QE models are supervised: they require human-labeled quality of some MT system output on some datasets for training, making them domain-dependent and MT-system-dependent. There has been research on unsupervised QE, which requires glass-box access to the MT systems, or parallel MT data to generate synthetic errors for training QE models. In this paper, we present Perturbation-based QE - a word-level Quality Estimation approach that works simply by analyzing MT system output on perturbed input source sentences. Our approach is unsupervised, explainable, and can evaluate any type of blackbox MT systems, including the currently prominent large language models (LLMs) with opaque internal processes. For language directions with no labeled QE data, our approach has similar or better performance than the zero-shot supervised approach on the WMT21 shared task. Our approach is better at detecting gender bias and word-sense-disambiguation errors in translation than supervised QE, indicating its robustness to out-of-domain usage. The performance gap is larger when detecting errors on a nontraditional translation-prompting LLM, indicating that our approach is more generalizable to different MT systems. We give examples demonstrating our approach's explainability power, where it shows which input source words have influence on a certain MT output word.

翻译：质量评估（QE）是预测机器翻译（MT）系统输出质量的任务，无需使用任何黄金标准翻译参考。当前最先进的QE模型是有监督的：它们需要某些数据集上MT系统输出的人工标注质量进行训练，这使得它们依赖领域和MT系统。已有研究探讨无监督QE，但这类方法需要访问MT系统的玻璃盒（即内部结构），或需要平行MT数据来生成合成错误以训练QE模型。本文提出基于扰动的QE——一种词级质量评估方法，仅通过分析MT系统在扰动输入源句上的输出来工作。我们的方法是无监督的、可解释的，并且能评估任何类型的黑盒MT系统，包括当前流行的、内部过程不透明的的大语言模型（LLMs）。在缺乏标注QE数据的语言方向上，我们的方法在WMT21共享任务上表现出与零样本有监督方法相似或更优的性能。与有监督QE相比，我们的方法在检测翻译中的性别偏见和词义消歧错误方面更优，表明其对领域外使用的鲁棒性。在检测非传统翻译提示的LLM的错误时，性能差距更大，表明我们的方法对不同MT系统更具泛化能力。我们通过示例展示了方法的可解释性，即能指出哪些输入源词对特定MT输出词产生影响。