Large Language Models (LLMs) have demonstrated considerable cross-lingual alignment and generalization ability. Current research primarily focuses on improving LLMs' cross-lingual generalization capabilities. However, there is still a lack of research on the intrinsic mechanisms of how LLMs achieve cross-lingual alignment. From the perspective of region partitioning, this paper conducts several investigations on the linguistic competence of LLMs. We discover a core region in LLMs that corresponds to linguistic competence, accounting for approximately 1% of the total model parameters. Removing this core region by setting parameters to zero results in a significant performance decrease across 30 different languages. Furthermore, this core region exhibits significant dimensional dependency, perturbations to even a single parameter on specific dimensions leading to a loss of linguistic competence. Moreover, we discover that distinct regions exist for different monolingual families, and disruption to these specific regions substantially reduces the LLMs' proficiency in those corresponding languages. Our research also indicates that freezing the core linguistic region during further pre-training can mitigate the issue of catastrophic forgetting (CF), a common occurrence observed during further pre-training of LLMs. Overall, exploring the LLMs' functional regions provides insights into the foundation of their intelligence.

翻译：大型语言模型（LLMs）已展现出显著的跨语言对齐能力和泛化能力。当前研究主要聚焦于提升LLMs的跨语言泛化性能，然而对其实现跨语言对齐的内在机制仍缺乏深入探究。本文从区域划分视角，对LLMs的语言能力展开多项研究。我们发现LLMs中存在一个与语言能力对应的核心区域，约占模型总参数的1%。将该核心区域参数置零后，模型在30种不同语言上的性能均出现显著下降。此外，该核心区域表现出显著的维度依赖性，即使对特定维度上单个参数的扰动也会导致语言能力丧失。进一步研究表明，不同单语族存在对应的独特区域，破坏这些特定区域会显著降低LLMs对应语言的处理能力。我们的研究还发现，在持续预训练过程中冻结核心语言区域，能够缓解LLMs在该阶段常见灾难性遗忘（CF）问题。总体而言，探索LLMs的功能区域为理解其智能基础提供了重要启示。