Large Language Models (LLMs) exhibit remarkable capabilities in handling multiple tasks across domains due to their emergent properties. These capabilities are further augmented during the Supervised Fine-Tuning (SFT) phase. Despite their potential, existing work mainly focuses on domain-specific enhancements during fine-tuning, the challenge of which lies in catastrophic forgetting of knowledge across other domains. In this study, we introduce VersaTune, a novel data composition framework designed for enhancing LLMs' overall multi-ability performances during fine-tuning. We categorize knowledge into distinct domains including law, medicine, finance, science, code. We begin with detecting the distribution of domain-specific knowledge within the base model, followed by the composition of training data that aligns with the model's existing knowledge distribution. During the fine-tuning process, weights of different domains are dynamically adjusted based on their learnable potential and forgetting degree. Experimental results demonstrate that VersaTune achieves significant improvements in multi-domain performance, with a 35.21% enhancement in comprehensive multi-domain tasks. Additionally, in scenarios where specific domain optimization is required, VersaTune reduces the degradation of performance in other domains by 38.77%, without compromising the target domain's training efficacy.

翻译：大语言模型（LLMs）凭借其涌现特性，在跨领域处理多项任务时展现出卓越能力。这些能力在监督微调（SFT）阶段得到进一步增强。尽管潜力巨大，现有工作主要集中于微调过程中的领域特定增强，其挑战在于可能导致其他领域知识的灾难性遗忘。本研究提出VersaTune——一种新颖的数据组合框架，旨在微调过程中全面提升LLMs的多领域综合能力。我们将知识划分为法律、医学、金融、科学、代码等不同领域。首先检测基础模型中领域特定知识的分布情况，随后构建与模型现有知识分布相匹配的训练数据组合。在微调过程中，根据不同领域的学习潜力与遗忘程度动态调整其权重。实验结果表明，VersaTune在多领域性能上取得显著提升，综合多领域任务性能增强35.21%。此外，在需要特定领域优化的场景中，VersaTune在不影响目标领域训练效果的前提下，将其他领域的性能衰减降低了38.77%。