In recent studies, the extensive utilization of large language models has underscored the importance of robust evaluation methodologies for assessing text generation quality and relevance to specific tasks. This has revealed a prevalent issue known as hallucination, an emergent condition in the model where generated text lacks faithfulness to the source and deviates from the evaluation criteria. In this study, we formally define hallucination and propose a framework for its quantitative detection in a zero-shot setting, leveraging our definition and the assumption that model outputs entail task and sample specific inputs. In detecting hallucinations, our solution achieves an accuracy of 0.78 in a model-aware setting and 0.61 in a model-agnostic setting. Notably, our solution maintains computational efficiency, requiring far less computational resources than other SOTA approaches, aligning with the trend towards lightweight and compressed models.

翻译：在近期研究中，大规模语言模型的广泛应用凸显了评估文本生成质量及其与特定任务相关性的稳健方法论的重要性。这揭示了一个普遍存在的问题——幻觉，即模型生成文本缺乏对源内容的忠实度且偏离评估标准的一种涌现状态。本研究正式定义了幻觉，并基于该定义及"模型输出蕴含任务与样本特定输入"的假设，提出了零样本场景下幻觉的定量检测框架。在幻觉检测中，我们的方法在模型感知场景下达到0.78的准确率，在模型无关场景下达到0.61的准确率。值得注意的是，该方案保持计算高效性，所需计算资源远低于其他最优方法，契合当前轻量化与压缩模型的发展趋势。