Supervised fine-tuning (SFT) is a crucial step for adapting large language models (LLMs) to downstream tasks. However, conflicting objectives across heterogeneous SFT tasks often induce the "seesaw effect": optimizing for one task may degrade performance on others, particularly when model parameters are updated indiscriminately. In this paper, we propose a principled approach to disentangle and isolate task-specific parameter regions, motivated by the hypothesis that parameter heterogeneity underlies cross-task interference. Specifically, we first independently fine-tune LLMs on diverse SFT tasks and identify each task's core parameter region as the subset of parameters exhibiting the largest updates. Tasks with highly overlapping core parameter regions are merged for joint training, while disjoint tasks are organized into different stages. During multi-stage SFT, core parameters acquired in prior tasks are frozen, thereby preventing overwriting by subsequent tasks. To verify the effectiveness of our method, we conducted intensive experiments on multiple public datasets. The results showed that our dynamic parameter isolation strategy consistently reduced data conflicts and achieved consistent performance improvements compared to multi-stage and multi-task tuning baselines.

翻译：监督微调（SFT）是将大型语言模型（LLMs）适配到下游任务的关键步骤。然而，异构SFT任务间相互冲突的目标常引发“跷跷板效应”：优化某一任务可能导致其他任务性能下降，尤其是在模型参数被无差别更新时。本文提出一种基于原则的方法来解耦并隔离任务特定的参数区域，其动机源于参数异质性构成跨任务干扰基础的假设。具体而言，我们首先在不同SFT任务上独立微调LLMs，并将每个任务的核心参数区域识别为更新幅度最大的参数子集。核心参数区域高度重叠的任务被合并进行联合训练，而不相关的任务则被组织到不同阶段。在多阶段SFT过程中，先前任务获得的核心参数被冻结，从而避免被后续任务覆盖。为验证方法的有效性，我们在多个公开数据集上进行了密集实验。结果表明，与多阶段和多任务微调基线相比，我们的动态参数隔离策略持续减少了数据冲突，并实现了稳定的性能提升。