Large language models (LLMs) have learned vast amounts of factual knowledge through self-supervised pre-training on large-scale corpora. Meanwhile, LLMs have also demonstrated excellent multilingual capabilities, which can express the learned knowledge in multiple languages. However, the knowledge storage mechanism in LLMs still remains mysterious. Some researchers attempt to demystify the factual knowledge in LLMs from the perspective of knowledge neurons, and subsequently discover language-agnostic knowledge neurons that store factual knowledge in a form that transcends language barriers. However, the preliminary finding suffers from two limitations: 1) High Uncertainty in Localization Results. Existing study only uses a prompt-based probe to localize knowledge neurons for each fact, while LLMs cannot provide consistent answers for semantically equivalent queries. Thus, it leads to inaccurate localization results with high uncertainty. 2) Lack of Analysis in More Languages. The study only analyzes language-agnostic knowledge neurons on English and Chinese data, without exploring more language families and languages. Naturally, it limits the generalizability of the findings. To address aforementioned problems, we first construct a new benchmark called Rephrased Multilingual LAMA (RML-LAMA), which contains high-quality cloze-style multilingual parallel queries for each fact. Then, we propose a novel method named Multilingual Integrated Gradients with Uncertainty Estimation (MATRICE), which quantifies the uncertainty across queries and languages during knowledge localization. Extensive experiments show that our method can accurately localize language-agnostic knowledge neurons. We also further investigate the role of language-agnostic knowledge neurons in cross-lingual knowledge editing, knowledge enhancement and new knowledge injection.

翻译：大语言模型（LLMs）通过在大规模语料上进行自监督预训练，已学习到海量事实性知识。同时，LLMs也展现出卓越的多语言能力，能够用多种语言表达所学知识。然而，LLMs中的知识存储机制仍不明确。部分研究者尝试从知识神经元的角度揭示LLMs中的事实性知识，并发现了以超越语言障碍的形式存储事实性知识的语言无关知识神经元。但该初步发现存在两方面局限：1）定位结果的高不确定性。现有研究仅使用基于提示的探针为每个事实定位知识神经元，而LLMs无法对语义等价的查询提供一致回答，导致定位结果不准确且不确定性高。2）缺乏对更多语言的分析。该研究仅基于英语和中文数据分析了语言无关知识神经元，未探索更多语系及语言，这自然限制了研究结论的普适性。为解决上述问题，我们首先构建了一个名为"改写多语言LAMA"（RML-LAMA）的新基准数据集，其中包含针对每个事实的高质量完形填空式多语言平行查询。随后，我们提出了一种名为"带不确定性估计的多语言集成梯度"（MATRICE）的新方法，该方法能在知识定位过程中量化跨查询与跨语言的不确定性。大量实验表明，我们的方法能够准确定位语言无关知识神经元。我们进一步探究了语言无关知识神经元在跨语言知识编辑、知识增强及新知识注入中的作用。