Hallucination detection is a critical step toward understanding the trustworthiness of modern language models (LMs). To achieve this goal, we re-examine existing detection approaches based on the self-consistency of LMs and uncover two types of hallucinations resulting from 1) question-level and 2) model-level, which cannot be effectively identified through self-consistency check alone. Building upon this discovery, we propose a novel sampling-based method, i.e., semantic-aware cross-check consistency (SAC$^3$) that expands on the principle of self-consistency checking. Our SAC$^3$ approach incorporates additional mechanisms to detect both question-level and model-level hallucinations by leveraging advances including semantically equivalent question perturbation and cross-model response consistency checking. Through extensive and systematic empirical analysis, we demonstrate that SAC$^3$ outperforms the state of the art in detecting both non-factual and factual statements across multiple question-answering and open-domain generation benchmarks.

翻译：幻觉检测是理解现代语言模型（LMs）可信性的关键步骤。为此，我们重新审视了基于LM自洽性的现有检测方法，并揭示了由1）问题级和2）模型级导致的两种幻觉类型，这些幻觉无法仅通过自洽性检查有效识别。基于此发现，我们提出了一种新颖的基于采样的方法，即语义感知交叉检查一致性（SAC$^3$），该方法扩展了自洽性检查的原则。我们的SAC$^3$方法通过利用包括语义等价问题扰动和跨模型响应一致性检查在内的技术进步，融入了额外的机制来检测问题级和模型级幻觉。通过广泛而系统的实证分析，我们证明SAC$^3$在多个问答和开放域生成基准测试中，在检测非事实性和事实性陈述方面均优于现有技术。