Pre-trained language models (PLMs) contain vast amounts of factual knowledge, but how the knowledge is stored in the parameters remains unclear. This paper delves into the complex task of understanding how factual knowledge is stored in multilingual PLMs, and introduces the Architecture-adapted Multilingual Integrated Gradients method, which successfully localizes knowledge neurons more precisely compared to current methods, and is more universal across various architectures and languages. Moreover, we conduct an in-depth exploration of knowledge neurons, leading to the following two important discoveries: (1) The discovery of Language-Independent Knowledge Neurons, which store factual knowledge in a form that transcends language. We design cross-lingual knowledge editing experiments, demonstrating that the PLMs can accomplish this task based on language-independent neurons; (2) The discovery of Degenerate Knowledge Neurons, a novel type of neuron showing that different knowledge neurons can store the same fact. Its property of functional overlap endows the PLMs with a robust mastery of factual knowledge. We design fact-checking experiments, proving that the degenerate knowledge neurons can help the PLMs to detect wrong facts. Experiments corroborate these findings, shedding light on the mechanisms of factual knowledge storage in multilingual PLMs, and contribute valuable insights to the field. The code is available at https://github.com/heng840/AMIG.

翻译：预训练语言模型（PLMs）包含大量事实性知识，但这些知识如何在参数中存储仍不清楚。本文深入探究了多语言PLMs中事实性知识存储的复杂任务，并提出了架构自适应多语言积分梯度方法。相比现有方法，该方法能更精准地定位知识神经元，且在不同架构和语言中具有更强的通用性。此外，我们对知识神经元进行了深入探索，获得了以下两项重要发现：(1) 发现语言无关知识神经元，它们以超越语言的形式存储事实性知识。我们设计了跨语言知识编辑实验，证明PLMs能够基于语言无关神经元完成该任务；(2) 发现退化知识神经元这一新型神经元，表明不同知识神经元可存储相同事实。其功能重叠特性赋予了PLMs对事实性知识的稳健掌握能力。我们设计了事实核查实验，证明退化知识神经元能帮助PLMs检测错误事实。实验结果印证了这些发现，揭示了多语言PLMs中事实性知识的存储机制，为该领域提供了宝贵见解。相关代码已公开于https://github.com/heng840/AMIG。