Pre-trained language models (PLMs) contain vast amounts of factual knowledge, but how the knowledge is stored in the parameters remains unclear. This paper delves into the complex task of understanding how factual knowledge is stored in multilingual PLMs, and introduces the Architecture-adapted Multilingual Integrated Gradients method, which successfully localizes knowledge neurons more precisely compared to current methods, and is more universal across various architectures and languages. Moreover, we conduct an in-depth exploration of knowledge neurons, leading to the following two important discoveries: (1) The discovery of Language-Independent Knowledge Neurons, which store factual knowledge in a form that transcends language. We design cross-lingual knowledge editing experiments, demonstrating that the PLMs can accomplish this task based on language-independent neurons; (2) The discovery of Degenerate Knowledge Neurons, a novel type of neuron showing that different knowledge neurons can store the same fact. Its property of functional overlap endows the PLMs with a robust mastery of factual knowledge. We design fact-checking experiments, proving that the degenerate knowledge neurons can help the PLMs to detect wrong facts. Experiments corroborate these findings, shedding light on the mechanisms of factual knowledge storage in multilingual PLMs, and contribute valuable insights to the field. The source code will be made publicly available for further research.

翻译：预训练语言模型（PLMs）蕴含海量事实性知识，但知识在参数中的存储机制尚不明确。本文深入探究多语言PLMs中事实性知识的存储方式，提出架构自适应多语言集成梯度方法（Architecture-adapted Multilingual Integrated Gradients method），该方法相较于现有技术能更精准定位知识神经元，且对不同架构和语言具有更强的普适性。我们进一步对知识神经元展开深度探索，取得两项重要发现：（1）发现语言无关性知识神经元——这些神经元以超越语言的形式存储事实性知识。通过设计跨语言知识编辑实验，证明PLMs可基于这些语言无关神经元完成该任务；（2）发现退化性知识神经元——这类新型神经元揭示不同知识神经元可存储同一事实。其功能重叠特性赋予PLMs对事实性知识的鲁棒掌握能力。我们设计事实检验实验，证明退化性知识神经元能辅助PLMs检测错误事实。实验结果验证了上述发现，揭示了多语言PLMs中事实性知识的存储机制，为该领域提供了重要见解。相关源代码将公开发布以支持后续研究。