Automatic evaluation of generated textual content presents an ongoing challenge within the field of NLP. Given the impressive capabilities of modern language models (LMs) across diverse NLP tasks, there is a growing trend to employ these models in creating innovative evaluation metrics for automated assessment of generation tasks. This paper investigates a pivotal question: Do language model-driven evaluation metrics inherently exhibit bias favoring texts generated by the same underlying language model? Specifically, we assess whether prominent LM-based evaluation metrics (e.g. BARTScore, T5Score, and GPTScore) demonstrate a favorable bias toward their respective underlying LMs in the context of summarization tasks. Our findings unveil a latent bias, particularly pronounced when such evaluation metrics are used in a reference-free manner without leveraging gold summaries. These results underscore that assessments provided by generative evaluation models can be influenced by factors beyond the inherent text quality, highlighting the necessity of developing more reliable evaluation protocols in the future.

翻译：生成文本内容的自动评估一直是自然语言处理领域持续面临的挑战。鉴于现代语言模型在各种自然语言处理任务中展现出的卓越能力，越来越多研究倾向于利用这些模型创建创新性的评估指标，以实现生成任务的自动化评估。本文探究了一个关键问题：基于语言模型的评估指标是否本质上存在偏向于同一底层语言模型生成文本的偏见？具体而言，我们评估了在文本摘要任务中，主流的基于语言模型的评估指标（如BARTScore、T5Score和GPTScore）是否对其各自底层语言模型表现出偏向性。我们的研究结果揭示了一种潜在偏见，这种偏见在评估指标以无参考方式（即不利用黄金摘要）使用时尤为显著。这些发现表明，生成式评估模型提供的评估结果可能受到文本固有质量之外因素的影响，这凸显了未来开发更可靠评估方案的必要性。