We introduce XFT, a simple yet powerful training scheme, by simply merging upcycled Mixture-of-Experts (MoE) to unleash the performance limit of instruction-tuned code Large Language Models (LLMs). While vanilla sparse upcycling fails to improve instruction tuning, XFT introduces a shared expert mechanism with a novel routing weight normalization strategy into sparse upcycling, which significantly boosts instruction tuning. After fine-tuning the upcycled MoE model, XFT introduces a learnable model merging mechanism to compile the upcycled MoE model back to a dense model, achieving upcycled MoE-level performance with only dense-model compute. By applying XFT to a 1.3B model, we create a new state-of-the-art tiny code LLM (<3B) with 67.1 and 64.6 pass@1 on HumanEval and HumanEval+ respectively. With the same data and model architecture, XFT improves supervised fine-tuning (SFT) by 13% on HumanEval+, along with consistent improvements from 2% to 13% on MBPP+, MultiPL-E, and DS-1000, demonstrating its generalizability. XFT is fully orthogonal to existing techniques such as Evol-Instruct and OSS-Instruct, opening a new dimension for improving code instruction tuning. Codes are available at https://github.com/ise-uiuc/xft.

翻译：我们提出XFT，一种简单而强大的训练方案，通过简单合并升级回收的专家混合模型（MoE），以释放指令调优代码大语言模型（LLMs）的性能极限。虽然原始稀疏升级回收未能改进指令调优，但XFT通过引入共享专家机制及新颖的路由权重归一化策略到稀疏升级回收中，显著提升了指令调优效果。在对升级回收的MoE模型进行微调后，XFT采用可学习的模型合并机制将升级回收的MoE模型编译回稠密模型，仅需稠密模型的计算量即可实现升级回收MoE级别的性能。将XFT应用于1.3B参数模型后，我们创造了新的最先进微型代码LLM（<3B），在HumanEval和HumanEval+上分别达到67.1和64.6的pass@1分数。在相同数据和模型架构下，XFT将监督微调（SFT）在HumanEval+上的性能提升13%，同时在MBPP+、MultiPL-E和DS-1000上实现2%至13%的稳定改进，证明了其泛化能力。XFT与现有技术（如Evol-Instruct和OSS-Instruct）完全正交，为改进代码指令调优开辟了新维度。代码发布于https://github.com/ise-uiuc/xft。