Abstractive text summarization has garnered increased interest as of late, in part due to the proliferation of large language models (LLMs). One of the most pressing problems related to generation of abstractive summaries is the need to reduce "hallucinations," information that was not included in the document being summarized, and which may be wholly incorrect. Due to this need, a wide array of metrics estimating consistency with the text being summarized have been proposed. We examine in particular a suite of unsupervised metrics for summary consistency, and measure their correlations with each other and with human evaluation scores in the wiki_bio_gpt3_hallucination dataset. We then compare these evaluations to models made from a simple linear ensemble of these metrics. We find that LLM-based methods outperform other unsupervised metrics for hallucination detection. We also find that ensemble methods can improve these scores even further, provided that the metrics in the ensemble have sufficiently similar and uncorrelated error rates. Finally, we present an ensemble method for LLM-based evaluations that we show improves over this previous SOTA.

翻译：抽象式文本摘要近年来因大型语言模型（LLMs）的普及而受到广泛关注。与抽象摘要生成相关的最紧迫问题之一是减少"幻觉"现象——即摘要中包含源文档未提及、甚至可能完全错误的信息。为应对这一需求，研究者提出了大量用于评估摘要与原文一致性的度量指标。本文重点研究一组无监督摘要一致性度量，并在wiki_bio_gpt3_hallucination数据集中分析这些度量之间的相关性及其与人工评估结果的关联。随后，我们将这些评估结果与基于简单线性集成方法构建的模型进行对比。实验发现：基于LLM的方法在幻觉检测上显著优于其他无监督度量；当集成中各度量具备充分相似且不相关的错误率时，集成方法可进一步提升检测性能。最后，我们提出一种面向LLM评估的集成方法，实验证明该方法超越了此前的最优水平（SOTA）。