Model merging has shown great promise at combining expert models, but the benefit of merging is unclear when merging ``generalist'' models trained on many tasks. We explore merging in the context of large (~100B) models, by recycling checkpoints that exhibit tradeoffs among different tasks. Such checkpoints are often created in the process of developing a frontier model, and many suboptimal ones are usually discarded. Given a pool of model checkpoints obtained from different training runs (e.g., different stages, objectives, hyperparameters, and data mixtures), which naturally show tradeoffs across different language capabilities (e.g., instruction following vs. code generation), we investigate whether merging can recycle such suboptimal models into a Pareto-optimal one. Our optimization algorithm tunes the weight of each checkpoint in a linear combination, resulting in a Pareto-optimal models that outperforms both individual models and merge-based baselines. Further analysis shows that good merges tend to include almost all checkpoints with non-zero weights, indicating that even seemingly bad initial checkpoints can contribute to good final merges.

翻译：模型合并技术在融合专家模型方面展现出巨大潜力，但当合并基于多任务训练的“通才”模型时，其效益尚不明确。本研究探索了在大型（约1000亿参数）模型背景下的合并方法，通过回收在各项任务间存在性能权衡的检查点来实现优化。此类检查点通常在开发前沿模型的过程中产生，其中大量次优检查点往往被弃用。给定一组从不同训练运行（例如不同训练阶段、目标函数、超参数和数据混合比例）获得的模型检查点池——这些检查点天然在不同语言能力（例如指令遵循与代码生成）间存在权衡——我们探究了合并技术能否将此类次优模型回收转化为帕累托最优模型。我们的优化算法通过调整线性组合中各检查点的权重，最终生成的帕累托最优模型在性能上超越了单个模型及基于合并的基线方法。进一步分析表明，优质合并结果倾向于以非零权重包含几乎所有检查点，这说明即使初始检查点看似欠佳，仍可能对最终合并产生积极贡献。