Translation Quality Estimation (QE) is the task of predicting the quality of machine translation (MT) output without any reference. This task has gained increasing attention as an important component in the practical applications of MT. In this paper, we first propose XLMRScore, which is a cross-lingual counterpart of BERTScore computed via the XLM-RoBERTa (XLMR) model. This metric can be used as a simple unsupervised QE method, while employing it results in two issues: firstly, the untranslated tokens leading to unexpectedly high translation scores, and secondly, the issue of mismatching errors between source and hypothesis tokens when applying the greedy matching in XLMRScore. To mitigate these issues, we suggest replacing untranslated words with the unknown token and the cross-lingual alignment of the pre-trained model to represent aligned words closer to each other, respectively. We evaluate the proposed method on four low-resource language pairs of WMT21 QE shared task, as well as a new English-Farsi test dataset introduced in this paper. Experiments show that our method could get comparable results with the supervised baseline for two zero-shot scenarios, i.e., with less than 0.01 difference in Pearson correlation, while outperforming unsupervised rivals in all the low-resource language pairs for above 8%, on average.

翻译：翻译质量评估（QE）是指在没有任何参考译文的情况下预测机器翻译（MT）输出质量的任务。该任务作为MT实际应用中的重要组成部分，已日益受到关注。本文首先提出XLMRScore，这是一种通过XLM-RoBERTa（XLMR）模型计算得到的BERTScore跨语言对应指标。该指标可作为简单的无监督QE方法使用，但其应用存在两个问题：一是未翻译词元导致翻译得分异常偏高，二是在XLMRScore的贪心匹配过程中，源语言与假设词元之间出现错配错误。为缓解这些问题，我们分别提出了将未翻译词替换为未知词元，以及利用预训练模型的跨语言对齐使对齐词在表示空间更接近的方法。我们在WMT21 QE共享任务的四个低资源语言对以及本文新引入的英语-波斯语测试数据集上评估了所提方法。实验表明，我们的方法在两个零样本场景下可与监督基线方法取得相当的结果（皮尔逊相关系数差异小于0.01），同时在所有低资源语言对上的平均性能较无监督竞争对手高出8%以上。