Recently proposed BERT-based evaluation metrics for text generation perform well on standard benchmarks but are vulnerable to adversarial attacks, e.g., relating to information correctness. We argue that this stems (in part) from the fact that they are models of semantic similarity. In contrast, we develop evaluation metrics based on Natural Language Inference (NLI), which we deem a more appropriate modeling. We design a preference-based adversarial attack framework and show that our NLI based metrics are much more robust to the attacks than the recent BERT-based metrics. On standard benchmarks, our NLI based metrics outperform existing summarization metrics, but perform below SOTA MT metrics. However, when combining existing metrics with our NLI metrics, we obtain both higher adversarial robustness (15%-30%) and higher quality metrics as measured on standard benchmarks (+5% to 30%).

翻译：近期提出的基于BERT的文本生成评估指标在标准基准测试中表现良好，但容易受到对抗攻击（例如与信息正确性相关的攻击）。我们认为，这（部分）源于这些指标本质上是语义相似度模型。相比之下，我们开发了基于自然语言推理（NLI）的评估指标，认为这是一种更合适的建模方式。我们设计了一个基于偏好的对抗攻击框架，并证明基于NLI的指标比近期基于BERT的指标对攻击具有更强的稳健性。在标准基准测试中，基于NLI的指标优于现有摘要评估指标，但略低于当前最优的机器翻译指标。然而，将现有指标与我们的NLI指标结合使用时，不仅能获得更高的对抗稳健性（提升15%-30%），还能在标准基准测试中获得质量更高的指标（提升5%-30%）。