The quality of texts generated by natural language generation (NLG) systems is hard to measure automatically. Conventional reference-based metrics, such as BLEU and ROUGE, have been shown to have relatively low correlation with human judgments, especially for tasks that require creativity and diversity. Recent studies suggest using large language models (LLMs) as reference-free metrics for NLG evaluation, which have the benefit of being applicable to new tasks that lack human references. However, these LLM-based evaluators still have lower human correspondence than medium-size neural evaluators. In this work, we present G-Eval, a framework of using large language models with chain-of-thoughts (CoT) and a form-filling paradigm, to assess the quality of NLG outputs. We experiment with two generation tasks, text summarization and dialogue generation. We show that G-Eval with GPT-4 as the backbone model achieves a Spearman correlation of 0.514 with human on summarization task, outperforming all previous methods by a large margin. We also propose preliminary analysis on the behavior of LLM-based evaluators, and highlight the potential issue of LLM-based evaluators having a bias towards the LLM-generated texts.

翻译：自然语言生成（NLG）系统所生成文本的质量难以自动衡量。传统的基于参考指标的评估方法（如BLEU和ROUGE）已被证明与人工判断的相关性较低，尤其对于需要创造性和多样性的任务。近期研究表明，将大型语言模型（LLMs）作为无参考指标用于NLG评估具有优势，可适用于缺乏人工参考的新任务。然而，这些基于LLM的评估器与人类判断的一致性仍低于中等规模的神经评估器。本研究提出G-Eval框架，采用结合思维链（CoT）与表单填充范式的大型语言模型来评估NLG输出质量。我们在文本摘要和对话生成这两类生成任务上进行实验，结果表明：以GPT-4作为骨干模型的G-Eval在文本摘要任务上与人类的Spearman相关系数达到0.514，以显著优势超越所有现有方法。此外，我们初步分析了基于LLM的评估器的行为特征，并指出了这类评估器可能对LLM生成文本存在偏倚的潜在问题。