A prominent weakness of modern language models (LMs) is their tendency to generate factually incorrect text, which hinders their usability. A natural question is whether such factual errors can be detected automatically. Inspired by truth-seeking mechanisms in law, we propose a factuality evaluation framework for LMs that is based on cross-examination. Our key idea is that an incorrect claim is likely to result in inconsistency with other claims that the model generates. To discover such inconsistencies, we facilitate a multi-turn interaction between the LM that generated the claim and another LM (acting as an examiner) which introduces questions to discover inconsistencies. We empirically evaluate our method on factual claims made by multiple recent LMs on four benchmarks, finding that it outperforms existing methods and baselines, often by a large gap. Our results demonstrate the potential of using interacting LMs for capturing factual errors.

翻译：现代语言模型的一个显著弱点是倾向于生成事实不正确的文本，这阻碍了其可用性。一个自然的问题是，此类事实错误能否被自动检测。受法律中求实机制的启发，我们提出了一种基于交叉检验的语言模型事实性评估框架。其核心思想是：一个不正确的声明很可能与模型生成的其他声明产生不一致。为发现此类不一致，我们促成了生成该声明的语言模型与另一个（扮演检验者角色的）语言模型之间的多轮交互，后者通过提问来揭示不一致性。我们在四个基准上对多个最新语言模型所作的事实声明进行了实证评估，发现我们的方法在性能上大幅优于现有方法和基线。研究结果展现了利用交互式语言模型捕捉事实错误的潜力。