Recently proposed BERT-based evaluation metrics for text generation perform well on standard benchmarks but are vulnerable to adversarial attacks, e.g., relating to information correctness. We argue that this stems (in part) from the fact that they are models of semantic similarity. In contrast, we develop evaluation metrics based on Natural Language Inference (NLI), which we deem a more appropriate modeling. We design a preference-based adversarial attack framework and show that our NLI based metrics are much more robust to the attacks than the recent BERT-based metrics. On standard benchmarks, our NLI based metrics outperform existing summarization metrics, but perform below SOTA MT metrics. However, when combining existing metrics with our NLI metrics, we obtain both higher adversarial robustness (15%-30%) and higher quality metrics as measured on standard benchmarks (+5% to 30%).

翻译：最近提出的基于BERT的文本生成评估指标在标准基准测试中表现良好，但容易受到对抗性攻击，例如与信息正确性相关的攻击。我们认为，这（部分）源于这些指标本质上是语义相似性模型。相比之下，我们开发了基于自然语言推理（NLI）的评估指标，我们认为这是一种更合适的建模方法。我们设计了一个基于偏好的对抗性攻击框架，并证明基于NLI的指标比最近的基于BERT的指标对攻击更具稳健性。在标准基准测试中，我们的基于NLI的指标优于现有的摘要评估指标，但低于最先进的机器翻译指标。然而，当将现有指标与我们的NLI指标结合使用时，我们获得了更高的对抗稳健性（15%-30%），以及在标准基准测试中衡量的更高质量指标（+5%至30%）。