Recently proposed BERT-based evaluation metrics for text generation perform well on standard benchmarks but are vulnerable to adversarial attacks, e.g., relating to information correctness. We argue that this stems (in part) from the fact that they are models of semantic similarity. In contrast, we develop evaluation metrics based on Natural Language Inference (NLI), which we deem a more appropriate modeling. We design a preference-based adversarial attack framework and show that our NLI based metrics are much more robust to the attacks than the recent BERT-based metrics. On standard benchmarks, our NLI based metrics outperform existing summarization metrics, but perform below SOTA MT metrics. However, when combining existing metrics with our NLI metrics, we obtain both higher adversarial robustness (15%-30%) and higher quality metrics as measured on standard benchmarks (+5% to 30%).

翻译：近年来提出的基于BERT的文本生成评估指标在标准基准测试上表现良好，但在对抗攻击（尤其是涉及信息正确性的攻击）下易受干扰。我们认为，这一缺陷（部分）源于此类指标本质上是语义相似性模型。为此，我们开发了基于自然语言推理（NLI）的评估指标——这被视为更合适的建模范式。我们设计了一个基于偏好的对抗攻击框架，证明基于NLI的指标比最新的BERT指标具有更强的抗攻击鲁棒性。在标准基准测试中，我们的NLI指标优于现有摘要评估指标，但在机器翻译指标上低于当前最优水平。然而，将现有指标与NLI指标结合后，我们既获得了更高的对抗鲁棒性（提升15%-30%），又在标准基准测试中实现了更高质量的指标（提升5%至30%）。