Underlying data distributions of natural language, programming code, and mathematical symbols vary vastly, presenting a complex challenge for large language models (LLMs) that strive to achieve high performance across all three domains simultaneously. Achieving a very high level of proficiency for an LLM within a specific domain often requires extensive training with relevant corpora, which is typically accompanied by a sacrifice in performance in other domains. In this paper, we propose to fuse models that are already highly-specialized directly. The proposed fusing framework, UltraFuser, consists of three distinct specialists that are already sufficiently trained on language, coding, and mathematics. A token-level gating mechanism is introduced to blend the specialists' outputs. A two-stage training strategy accompanied by balanced sampling is designed to ensure stability. To effectively train the fused model, we further construct a high-quality supervised instruction tuning dataset, UltraChat 2, which includes text, code, and mathematical content. This dataset comprises approximately 300,000 instructions and covers a wide range of topics in each domain. Experiments show that our model could simultaneously achieve mastery of the three crucial domains.

翻译：自然语言、编程代码与数学符号的底层数据分布存在显著差异，这对试图同时在三个领域实现高性能的大型语言模型（LLMs）构成了复杂挑战。要使LLM在特定领域达到极高熟练度，通常需要大量相关语料进行训练，但这往往伴随着其他领域性能的牺牲。本文提出直接融合已高度专业化的模型。所提出的融合框架UltraFuser包含三个已在语言、编码和数学领域充分训练的独立专家模型，并引入令牌级门控机制来混合专家输出。为保障稳定性，我们设计了结合平衡采样的两阶段训练策略。为有效训练融合模型，我们进一步构建了包含文本、代码和数学内容的高质量监督指令微调数据集UltraChat 2，该数据集包含约30万条指令，覆盖各领域广泛主题。实验表明，我们的模型能同时实现对这三个关键领域的精通。