Large-scale pretrained models, particularly Large Language Models (LLMs), have exhibited remarkable capabilities in handling multiple tasks across domains due to their emergent properties. These capabilities are further augmented during the Supervised Fine-Tuning (SFT) phase. Despite their potential, existing work mainly focuses on domain-specific enhancements during fine-tuning, the challenge of which lies in catastrophic forgetting of knowledge across other domains. In this study, we introduce VersaTune, a novel data composition framework designed for enhancing LLMs' overall multi-ability performances during training. We categorize knowledge into distinct domains including law, medicine, finance, science, code, etc. We begin with detecting the distribution of domain-specific knowledge within the base model, followed by the training data composition that aligns with the model's existing knowledge distribution. During the training process, domain weights are dynamically adjusted based on their learnable potential and forgetting degree. Experimental results demonstrate that VersaTune achieves significant improvements in multi-domain performance, with an 35.21% enhancement in comprehensive multi-domain tasks. Additionally, in scenarios where specific domain optimization is required, VersaTune reduces the degradation of performance in other domains by 38.77%, without compromising the target domain's training efficacy.

翻译：大规模预训练模型，特别是大语言模型（LLM），凭借其涌现特性，在跨领域处理多种任务方面展现出卓越能力。这些能力在监督微调（SFT）阶段得到进一步增强。尽管潜力巨大，现有工作主要集中于微调过程中的领域特定增强，其挑战在于可能导致对其他领域知识的灾难性遗忘。本研究提出VersaTune，一种新颖的数据组合框架，旨在训练过程中提升LLM的整体多能力表现。我们将知识划分为不同领域，包括法律、医学、金融、科学、代码等。首先检测基础模型中领域特定知识的分布，随后构建与模型现有知识分布相匹配的训练数据组合。在训练过程中，根据各领域的学习潜力和遗忘程度动态调整其权重。实验结果表明，VersaTune在多领域性能上取得了显著提升，在综合性多领域任务中性能提高了35.21%。此外，在需要特定领域优化的场景下，VersaTune将其他领域的性能下降降低了38.77%，同时不影响目标领域的训练效果。