With Large Language Models (LLMs) being widely used across various tasks, detecting errors in their responses is increasingly crucial. However, little research has been conducted on error detection of LLM responses. Collecting error annotations on LLM responses is challenging due to the subjective nature of many NLP tasks, and thus previous research focuses on tasks of little practical value (e.g., word sorting) or limited error types (e.g., faithfulness in summarization). This work introduces ReaLMistake, the first error detection benchmark consisting of objective, realistic, and diverse errors made by LLMs. ReaLMistake contains three challenging and meaningful tasks that introduce objectively assessable errors in four categories (reasoning correctness, instruction-following, context-faithfulness, and parameterized knowledge), eliciting naturally observed and diverse errors in responses of GPT-4 and Llama 2 70B annotated by experts. We use ReaLMistake to evaluate error detectors based on 12 LLMs. Our findings show: 1) Top LLMs like GPT-4 and Claude 3 detect errors made by LLMs at very low recall, and all LLM-based error detectors perform much worse than humans. 2) Explanations by LLM-based error detectors lack reliability. 3) LLMs-based error detection is sensitive to small changes in prompts but remains challenging to improve. 4) Popular approaches to improving LLMs, including self-consistency and majority vote, do not improve the error detection performance. Our benchmark and code are provided at https://github.com/psunlpgroup/ReaLMistake.

翻译：随着大语言模型（LLMs）被广泛应用于各类任务，检测其响应中的错误变得日益重要。然而，目前针对LLM响应错误检测的研究尚不充分。由于许多自然语言处理任务具有主观性，收集LLM响应的错误标注颇具挑战，因此现有研究多聚焦于实践价值较低的任务（如词语排序）或有限的错误类型（如摘要忠实性）。本文提出ReaLMistake——首个由LLM产生的客观、真实且多样化的错误检测基准。ReaLMistake包含三个具有挑战性和实际意义的任务，涵盖四类可客观评估的错误（推理正确性、指令遵循性、上下文忠实性和参数化知识），并提取了GPT-4和Llama 2 70B响应中自然出现且经专家标注的多样化错误。我们利用ReaLMistake评估了基于12个LLM的错误检测器。研究发现：1）GPT-4和Claude 3等顶尖LLM在检测LLM错误时召回率极低，所有基于LLM的错误检测器性能均远低于人类；2）基于LLM的错误检测器提供的解释缺乏可靠性；3）基于LLM的错误检测对提示词的微小变化敏感，但改进难度较大；4）提升LLM性能的常用方法（如自一致性和多数投票）未能改善错误检测能力。基准测试和代码已开源至https://github.com/psunlpgroup/ReaLMistake。