Large language models underestimate the impact of negations on how much they change the meaning of a sentence. Therefore, learned evaluation metrics based on these models are insensitive to negations. In this paper, we propose NegBLEURT, a negation-aware version of the BLEURT evaluation metric. For that, we designed a rule-based sentence negation tool and used it to create the CANNOT negation evaluation dataset. Based on this dataset, we fine-tuned a sentence transformer and an evaluation metric to improve their negation sensitivity. Evaluating these models on existing benchmarks shows that our fine-tuned models outperform existing metrics on the negated sentences by far while preserving their base models' performances on other perturbations.

翻译：大型语言模型低估了否定词对句子意义改变的影响。因此，基于这些模型习得的评估指标对否定不敏感。本文提出NegBLEURT，一种具备否定感知能力的BLEURT评估指标版本。为此，我们设计了一套基于规则的句子否定工具，并利用该工具创建了CANNOT否定评估数据集。基于该数据集，我们对句子Transformer模型和评估指标进行微调，以提升其对否定现象的敏感度。在现有基准上的评估表明，我们的微调模型在否定句上的表现远超现有指标，同时在其他扰动任务上保持了基础模型的性能。