Large language models (LLMs) have achieved unprecedented performance in various applications, yet their evaluation remains a critical issue. Existing hallucination benchmarks are either static or lack adjustable complexity for thorough analysis. We contend that utilizing existing relational databases is a promising approach for constructing benchmarks due to their accurate knowledge description via functional dependencies. We propose ERBench to automatically convert any relational database into a benchmark based on the entity-relationship (ER) model. Our key idea is to construct questions using the database schema, records, and functional dependencies such that they can be automatically verified. In addition, we use foreign key constraints to join relations and construct multihop questions, which can be arbitrarily complex and used to debug the intermediate answers of LLMs. Finally, ERBench supports continuous evaluation, multimodal questions, and various prompt engineering techniques. In our experiments, we construct an LLM benchmark using databases of multiple domains and make an extensive comparison of contemporary LLMs. We observe that better LLMs like GPT-4 can handle a larger variety of question types, but are by no means perfect. Also, correct answers do not necessarily imply correct rationales, which is an important evaluation that ERBench does better than other benchmarks for various question types. Code is available at https: //github.com/DILAB-KAIST/ERBench.

翻译：大语言模型（LLMs）在各类应用中取得了前所未有的性能，但其评估仍是一个关键问题。现有幻觉基准测试要么是静态的，要么缺乏可调节的复杂度以进行深入分析。我们认为，利用现有关系数据库构建基准测试具有前景，因为功能依赖关系能够准确描述知识。我们提出ERBench，它基于实体关系（ER）模型，能够自动将任意关系数据库转换为基准测试。我们的核心思想是利用数据库模式、记录和功能依赖关系构造问题，使问题可被自动验证。此外，我们使用外键约束连接关系并构造多跳问题，这些问题可具有任意复杂度，并用于调试LLMs的中间答案。最后，ERBench支持持续评估、多模态问题及多种提示工程技术。在实验中，我们利用多个领域的数据库构建了LLM基准测试，并对当代LLMs进行了广泛比较。我们观察到，GPT-4等更优的LLM能处理更多样化的问题类型，但远非完美。此外，正确答案并不必然意味着正确推理过程，而ERBench在各类问题上对此的评估优于其他基准测试。代码参见 https://github.com/DILAB-KAIST/ERBench。