Large language models (LLMs) demonstrate remarkable performance across a spectrum of languages. In this work, we delve into the question: How do LLMs handle multilingualism? We introduce a framework that depicts LLMs' processing of multilingual inputs: In the first several layers, LLMs understand the question, converting multilingual inputs into English to facilitate the task-solving phase. In the intermediate layers, LLMs engage in problem-solving by thinking in English and incorporating multilingual knowledge to obtain factual content, leveraging the self-attention and feed-forward structures, respectively. In the last several layers, LLMs generate responses that align with the original language of the query. In addition, we investigate the existence of language-specific neurons when processing a certain language. To detect neurons activated by the input language, even without labels, we innovatively design a Parallel Language specific Neuron Detection ($\texttt{PLND}$) method that effectively measures the significance of neurons when handling multilingual inputs. By comprehensive ablation analysis through deactivating neurons of different layers and structures, we verify the framework that we propose. Additionally, we demonstrate that we can utilize such a framework to effectively enhance the multilingual ability with much less training effort.

翻译：大型语言模型（LLMs）在多种语言上展现出卓越的性能。本文深入探讨以下问题：大型语言模型如何处理多语言输入？我们提出一个框架，描绘LLMs处理多语言输入的流程：在最初几层，LLMs理解问题，将多语言输入转换为英语以促进问题解决阶段；在中间层，LLMs通过用英语思考并结合多语言知识获取事实内容，分别利用自注意力机制和前馈结构；在最后几层，LLMs生成与查询原始语言一致的响应。此外，我们研究了处理特定语言时是否存在语言特异性神经元。为检测由输入语言激活的神经元（即使没有标签），我们创新性地设计了并行语言特异性神经元检测方法（$\texttt{PLND}$），该方法能有效衡量神经元在处理多语言输入时的重要性。通过停用不同层和结构的神经元进行全面的消融分析，我们验证了所提出的框架。此外，我们证明可以利用该框架以更少的训练成本有效增强多语言能力。