Strongly human-correlated evaluation metrics serve as an essential compass for the development and improvement of generation models and must be highly reliable and robust. Recent embedding-based neural text evaluation metrics, such as COMET for translation tasks, are widely used in both research and development fields. However, there is no guarantee that they yield reliable evaluation results due to the black-box nature of neural networks. To raise concerns about the reliability and safety of such metrics, we propose a method for finding a single adversarial text in the discrete space that is consistently evaluated as high-quality, regardless of the test cases, to identify the vulnerabilities in evaluation metrics. The single hub text found with our method achieved 79.1 COMET% and 67.8 COMET% in the WMT'24 English-to-Japanese (En--Ja) and English-to-German (En--De) translation tasks, respectively, outperforming translations generated individually for each source sentence by using M2M100, a general translation model. Furthermore, we also confirmed that the hub text found with our method generalizes across multiple language pairs such as Ja--En and De--En.

翻译：与人类评价高度相关的评估指标作为生成模型发展与改进的关键指南，必须具备高度的可靠性与鲁棒性。近年来基于嵌入的神经文本评估指标（例如面向翻译任务的COMET）已在研究与应用领域得到广泛使用。然而，由于神经网络的黑箱特性，无法保证这些指标始终产生可靠的评估结果。为揭示此类指标在可靠性与安全性方面存在的隐患，本文提出一种在离散空间中寻找单对抗性文本的方法：该文本无论面对何种测试用例，均能持续获得高质量评价，从而暴露评估指标的内在脆弱性。通过本方法发现的单枢纽文本在WMT'24英日（En--Ja）与英德（En--De）翻译任务中分别达到79.1 COMET%与67.8 COMET%的评分，甚至超越了使用通用翻译模型M2M100为每个源语句单独生成的译文。此外，实验证实本方法发现的枢纽文本可泛化至日英（Ja--En）与德英（De--En）等多语言对场景。