The quality of texts generated by natural language generation (NLG) systems is hard to measure automatically. Conventional reference-based metrics, such as BLEU and ROUGE, have been shown to have relatively low correlation with human judgments, especially for tasks that require creativity and diversity. Recent studies suggest using large language models (LLMs) as reference-free metrics for NLG evaluation, which have the benefit of being applicable to new tasks that lack human references. However, these LLM-based evaluators still have lower human correspondence than medium-size neural evaluators. In this work, we present GPTEval, a framework of using large language models with chain-of-thoughts (CoT) and a form-filling paradigm, to assess the quality of NLG outputs. We experiment with two generation tasks, text summarization and dialogue generation. We show that GPTEval with GPT-4 as the backbone model achieves a Spearman correlation of 0.514 with human on summarization task, outperforming all previous methods by a large margin. We also propose preliminary analysis on the behavior of LLM-based evaluators, and highlight the potential issue of LLM-based evaluators having a bias towards the LLM-generated texts.

翻译：自然语言生成（NLG）系统生成文本的质量难以自动衡量。传统的基于参考的指标（如BLEU和ROUGE）已被证明与人类判断的相关性较低，尤其在需要创造性和多样性的任务中表现不佳。近期研究表明，将大型语言模型（LLMs）用作无参考的NLG评估指标具有优势，可适用于缺乏人类参考的新任务。然而，这些基于LLM的评估器与人类判断的一致性仍低于中等规模神经评估器。本研究提出GPTEval框架，该框架利用大型语言模型结合思维链（CoT）与表单填充范式来评估NLG输出质量。我们在文本摘要和对话生成两个生成任务上进行实验，结果表明，以GPT-4作为骨干模型的GPTEval在摘要任务上达到0.514的Spearman相关系数，大幅超越以往所有方法。我们进一步对基于LLM的评估器行为进行了初步分析，并指出其可能存在的偏向LLM生成文本的潜在问题。