Recently proposed BERT-based evaluation metrics for text generation perform well on standard benchmarks but are vulnerable to adversarial attacks, e.g., relating to information correctness. We argue that this stems (in part) from the fact that they are models of semantic similarity. In contrast, we develop evaluation metrics based on Natural Language Inference (NLI), which we deem a more appropriate modeling. We design a preference-based adversarial attack framework and show that our NLI based metrics are much more robust to the attacks than the recent BERT-based metrics. On standard benchmarks, our NLI based metrics outperform existing summarization metrics, but perform below SOTA MT metrics. However, when combining existing metrics with our NLI metrics, we obtain both higher adversarial robustness (15%-30%) and higher quality metrics as measured on standard benchmarks (+5% to 30%).

翻译：近期提出的基于BERT的文本生成评估指标在标准基准测试中表现良好，但存在对抗攻击（例如与信息正确性相关的攻击）下的脆弱性。我们认为这（部分）源于这些指标本质上是语义相似度模型。相比之下，我们开发了基于自然语言推理（NLI）的评估指标，认为这是更恰当的建模方式。我们设计了一个基于偏好的对抗攻击框架，实验表明，与近期基于BERT的指标相比，我们的基于NLI的指标对攻击具有更强的稳健性。在标准基准测试中，我们的基于NLI的指标优于现有摘要评估指标，但低于当前最先进的机器翻译指标。然而，当现有指标与我们的NLI指标结合使用时，我们既获得了更高的对抗稳健性（15%-30%），也获得了标准基准测试中更高质量的评估指标（提升幅度为5%-30%）。