In this paper, we establish a benchmark named HalluQA (Chinese Hallucination Question-Answering) to measure the hallucination phenomenon in Chinese large language models. HalluQA contains 450 meticulously designed adversarial questions, spanning multiple domains, and takes into account Chinese historical culture, customs, and social phenomena. During the construction of HalluQA, we consider two types of hallucinations: imitative falsehoods and factual errors, and we construct adversarial samples based on GLM-130B and ChatGPT. For evaluation, we design an automated evaluation method using GPT-4 to judge whether a model output is hallucinated. We conduct extensive experiments on 24 large language models, including ERNIE-Bot, Baichuan2, ChatGLM, Qwen, SparkDesk and etc. Out of the 24 models, 18 achieved non-hallucination rates lower than 50%. This indicates that HalluQA is highly challenging. We analyze the primary types of hallucinations in different types of models and their causes. Additionally, we discuss which types of hallucinations should be prioritized for different types of models.

翻译：本文构建了一个名为HalluQA（中文幻觉问答）的基准数据集，用于评估中文大型语言模型中的幻觉现象。HalluQA包含450个精心设计的对抗性问题，涵盖多个领域，并考虑了中国的历史文化、风俗习惯和社会现象。在构建HalluQA时，我们考虑了两种类型的幻觉：模仿性错误和事实性错误，并基于GLM-130B和ChatGPT构建了对抗性样本。在评估方面，我们设计了一种利用GPT-4的自动化评估方法，用于判断模型输出是否存在幻觉。我们对包括ERNIE-Bot、Baichuan2、ChatGLM、Qwen、SparkDesk等在内的24个大型语言模型进行了广泛实验。在这24个模型中，有18个模型的非幻觉率低于50%。这表明HalluQA具有极高的挑战性。我们分析了不同类型模型中幻觉的主要类型及其成因。此外，我们还讨论了不同类型模型应优先处理哪些类型的幻觉。