Reasoning capabilities are crucial for Large Language Models (LLMs), yet a notable gap exists between English and non-English languages. To bridge this disparity, some works fine-tune LLMs to relearn reasoning capabilities in non-English languages, while others replace non-English inputs with an external model's outputs such as English translation text to circumvent the challenge of LLM understanding non-English. Unfortunately, these methods often underutilize the built-in skilled reasoning and useful language understanding capabilities of LLMs. In order to better utilize the minds of reasoning and language understanding in LLMs, we propose a new method, namely MindMerger, which merges LLMs with the external language understanding capabilities from multilingual models to boost the multilingual reasoning performance. Furthermore, a two-step training scheme is introduced to first train to embeded the external capabilities into LLMs and then train the collaborative utilization of the external capabilities and the built-in capabilities in LLMs. Experiments on three multilingual reasoning datasets and a language understanding dataset demonstrate that MindMerger consistently outperforms all baselines, especially in low-resource languages. Without updating the parameters of LLMs, the average accuracy improved by 6.7% and 8.0% across all languages and low-resource languages on the MGSM dataset, respectively.

翻译：推理能力对于大语言模型至关重要，然而英语与非英语语言之间存在显著差距。为弥合这一差异，部分研究通过微调大语言模型使其重新学习非英语语言的推理能力，另一些研究则通过外部模型（如英语翻译文本）替换非英语输入，以规避大语言模型理解非英语的挑战。遗憾的是，这些方法往往未能充分利用大语言模型内置的熟练推理能力和有效的语言理解能力。为更好地整合大语言模型中的推理思维与语言理解能力，本文提出一种新方法——MindMerger，该方法通过融合大语言模型与多语言模型的外部语言理解能力，以提升多语言推理性能。此外，我们引入两阶段训练方案：首先训练将外部能力嵌入大语言模型，随后训练外部能力与模型内置能力的协同运用。在三个多语言推理数据集和一个语言理解数据集上的实验表明，MindMerger始终优于所有基线方法，尤其在低资源语言中表现突出。在不更新大语言模型参数的情况下，其在MGSM数据集上所有语言和低资源语言的平均准确率分别提升了6.7%和8.0%。