Large language models (LLMs) often make factually incorrect responses despite their success in various applications. In this paper, we hypothesize that relying heavily on simple co-occurrence statistics of the pre-training corpora is one of the main factors that cause factual errors. Our results reveal that LLMs are vulnerable to the co-occurrence bias, defined as preferring frequently co-occurred words over the correct answer. Consequently, LLMs struggle to recall facts whose subject and object rarely co-occur in the pre-training dataset although they are seen during finetuning. We show that co-occurrence bias remains despite scaling up model sizes or finetuning. Therefore, we suggest finetuning on a debiased dataset to mitigate the bias by filtering out biased samples whose subject-object co-occurrence count is high. Although debiased finetuning allows LLMs to memorize rare facts in the training set, it is not effective in recalling rare facts unseen during finetuning. Further research in mitigation will help build reliable language models by preventing potential errors. The code is available at \url{https://github.com/CheongWoong/impact_of_cooccurrence}.

翻译：大型语言模型（LLMs）在各种应用中取得了成功，但其回答常存在事实性错误。本文假设，过度依赖预训练语料库中的简单共现统计是导致事实错误的主要原因之一。研究结果表明，LLMs易受共现偏差影响——即倾向于选择高频共现词汇而非正确答案。因此，当预训练数据集中主语与宾语鲜少共现时，即便模型在微调阶段见过这些事实，仍难以正确回忆。我们证实，共现偏差不会因模型规模扩大或微调而消除。为此，建议通过去除主语-宾语共现频率较高的偏置样本来构建去偏数据集进行微调。虽然去偏微调能使LLMs记住训练集中的罕见事实，但对微调阶段未出现的罕见事实回忆效果仍不理想。进一步缓解该偏差的研究将为构建可靠语言模型、预防潜在错误提供支持。相关代码已开源至 \url{https://github.com/CheongWoong/impact_of_cooccurrence}。