Large language models (LLMs) have been widely studied for their ability to store and utilize positive knowledge. However, negative knowledge, such as "lions don't live in the ocean", is also ubiquitous in the world but rarely mentioned explicitly in the text. What do LLMs know about negative knowledge? This work examines the ability of LLMs to negative commonsense knowledge. We design a constrained keywords-to-sentence generation task (CG) and a Boolean question-answering task (QA) to probe LLMs. Our experiments reveal that LLMs frequently fail to generate valid sentences grounded in negative commonsense knowledge, yet they can correctly answer polar yes-or-no questions. We term this phenomenon the belief conflict of LLMs. Our further analysis shows that statistical shortcuts and negation reporting bias from language modeling pre-training cause this conflict.

翻译：大语言模型（LLMs）因其存储和运用正面知识的能力而被广泛研究。然而，负面知识（如“狮子不在海洋中生活”）同样普遍存在于世界中，却很少在文本中被明确提及。LLMs对负面知识了解多少？本研究检验了LLMs处理负面常识知识的能力。我们设计了一个受约束的关键词到句子生成任务（CG）和一个布尔问答任务（QA）来探究LLMs。实验揭示，LLMs经常无法生成基于负面常识知识的有效句子，但能正确回答极性的是非问题。我们将这一现象称为LLMs的信念冲突。进一步分析表明，语言模型预训练中的统计捷径和否定报告偏差导致了这一冲突。