Although neural-based machine translation evaluation metrics, such as COMET or BLEURT, have achieved strong correlations with human judgements, they are sometimes unreliable in detecting certain phenomena that can be considered as critical errors, such as deviations in entities and numbers. In contrast, traditional evaluation metrics, such as BLEU or chrF, which measure lexical or character overlap between translation hypotheses and human references, have lower correlations with human judgements but are sensitive to such deviations. In this paper, we investigate several ways of combining the two approaches in order to increase robustness of state-of-the-art evaluation methods to translations with critical errors. We show that by using additional information during training, such as sentence-level features and word-level tags, the trained metrics improve their capability to penalize translations with specific troublesome phenomena, which leads to gains in correlation with human judgments and on recent challenge sets on several language pairs.

翻译：尽管基于神经网络的机器翻译评估指标（如COMET或BLEURT）与人工判断已呈现强相关性，但在检测实体与数字偏差等可能被视为关键错误的现象时，其可靠性有时不足。相反，通过测量译文假设与人工参考间词汇或字符重叠的传统评估指标（如BLEU或chrF），虽与人工判断的相关性较低，却能敏锐捕捉此类偏差。本文探究了结合两种方法的多条路径，以期提升前沿评估方法对含关键错误译文的鲁棒性。研究表明，通过在训练过程中引入句子级特征与词级标签等额外信息，训练后的指标可增强对特定棘手现象译文的惩罚能力，从而在若干语言对的人工判断相关性指标及新近挑战性测试集上均获得性能提升。