In this paper, we establish a benchmark named HalluQA (Chinese Hallucination Question-Answering) to measure the hallucination phenomenon in Chinese large language models. HalluQA contains 450 meticulously designed adversarial questions, spanning multiple domains, and takes into account Chinese historical culture, customs, and social phenomena. During the construction of HalluQA, we consider two types of hallucinations: imitative falsehoods and factual errors, and we construct adversarial samples based on GLM-130B and ChatGPT. For evaluation, we design an automated evaluation method using GPT-4 to judge whether a model output is hallucinated. We conduct extensive experiments on 24 large language models, including ERNIE-Bot, Baichuan2, ChatGLM, Qwen, SparkDesk and etc. Out of the 24 models, 18 achieved non-hallucination rates lower than 50%. This indicates that HalluQA is highly challenging. We analyze the primary types of hallucinations in different types of models and their causes. Additionally, we discuss which types of hallucinations should be prioritized for different types of models.

翻译：本文构建了一个名为HalluQA（中文幻觉问答）的基准测试集，用于衡量中文大语言模型中的幻觉现象。HalluQA包含450个精心设计的对抗性问题，覆盖多个领域，并充分考虑了中国的历史文化、社会习俗及社会现象。在构建HalluQA过程中，我们考虑了两种幻觉类型：模仿性错误和事实性错误，并基于GLM-130B和ChatGPT构建了对抗样本。在评估阶段，我们设计了一种基于GPT-4的自动化评估方法，用于判断模型输出是否存在幻觉。我们对包括文心一言、百川2、智谱清言、通义千问、讯飞星火等在内的24个大语言模型进行了广泛实验。在这24个模型中，有18个模型的非幻觉率低于50%，这表明HalluQA具有极高的挑战性。我们分析了不同类型模型的主要幻觉类型及其成因，并进一步讨论了针对不同类型模型应优先解决的幻觉类型。