Large Language Models (LLMs) are widely used to evaluate natural language generation tasks as automated metrics. However, the likelihood, a measure of LLM's plausibility for a sentence, can vary due to superficial differences in sentences, such as word order and sentence structure. It is therefore possible that there might be a likelihood bias if LLMs are used for evaluation: they might overrate sentences with higher likelihoods while underrating those with lower likelihoods. In this paper, we investigate the presence and impact of likelihood bias in LLM-based evaluators. We also propose a method to mitigate the likelihood bias. Our method utilizes highly biased instances as few-shot examples for in-context learning. Our experiments in evaluating the data-to-text and grammatical error correction tasks reveal that several LLMs we test display a likelihood bias. Furthermore, our proposed method successfully mitigates this bias, also improving evaluation performance (in terms of correlation of models with human scores) significantly.

翻译：大语言模型（LLMs）被广泛用作自然语言生成任务的自动化评估指标。然而，作为衡量LLM对句子合理性的指标，似然性可能因句子表面差异（如词序和句子结构）而产生变化。因此，若将LLMs用于评估，可能存在似然性偏差：它们可能高估具有较高似然性的句子，同时低估似然性较低的句子。本文研究了基于LLM的评估器中似然性偏差的存在及其影响，并提出了一种缓解该偏差的方法。我们的方法利用高度偏差的实例作为上下文学习的少样本示例。在数据到文本和语法错误纠正任务的评估实验中，我们发现测试的多个LLMs均表现出似然性偏差。此外，所提出的方法成功缓解了该偏差，并显著提升了评估性能（以模型与人工评分的相关性衡量）。