Large-scale pretrained models, particularly Large Language Models (LLMs), have exhibited remarkable capabilities in handling multiple tasks across domains due to their emergent properties. These capabilities are further augmented during the Supervised Fine-Tuning (SFT) phase. Despite their potential, existing work mainly focuses on domain-specific enhancements during fine-tuning, the challenge of which lies in catastrophic forgetting of knowledge across other domains. In this study, we introduce VersaTune, a novel data composition framework designed for enhancing LLMs' overall multi-ability performances during training. We categorize knowledge into distinct domains including law, medicine, finance, science, code, etc. We begin with detecting the distribution of domain-specific knowledge within the base model, followed by the training data composition that aligns with the model's existing knowledge distribution. During the training process, domain weights are dynamically adjusted based on their learnable potential and forgetting degree. Experimental results demonstrate that VersaTune achieves significant improvements in multi-domain performance, with an 35.21% enhancement in comprehensive multi-domain tasks. Additionally, in scenarios where specific domain optimization is required, VersaTune reduces the degradation of performance in other domains by 38.77%, without compromising the target domain's training efficacy.

翻译：大规模预训练模型，尤其是大语言模型（LLMs），凭借其涌现特性，在跨领域处理多项任务方面展现出卓越能力。这些能力在监督微调（SFT）阶段得到进一步增强。尽管潜力巨大，现有工作主要集中于微调过程中的领域特定增强，其挑战在于可能导致对其他领域知识的灾难性遗忘。在本研究中，我们提出了VersaTune，一种新颖的数据组合框架，旨在训练过程中提升LLMs的整体多能力表现。我们将知识划分为不同领域，包括法律、医学、金融、科学、代码等。我们首先检测基础模型中领域特定知识的分布，随后构建与模型现有知识分布相匹配的训练数据组合。在训练过程中，领域权重根据其可学习潜力和遗忘程度进行动态调整。实验结果表明，VersaTune在多领域性能上取得了显著提升，在综合性多领域任务中性能提高了35.21%。此外，在需要特定领域优化的场景下，VersaTune在不影响目标领域训练效果的前提下，将其他领域的性能下降降低了38.77%。