Catastrophic forgetting emerges as a critical challenge when fine-tuning multi-modal large language models (MLLMs), where improving performance on unseen tasks often leads to a significant performance drop on the original tasks. This paper presents a comprehensive analysis of catastrophic forgetting in MLLMs and introduces a post-training adjustment method called Model Tailor. Our method primarily preserves the pre-trained parameters while replacing a small number ($\leq$ 10\%) of fine-tuned parameters, maintaining $\sim$ 99\% effectiveness on original tasks versus pre-training, and achieving $\sim$ 97\% on new tasks compared to standard fine-tuning. Specifically, we derive a sparse mask to identify the "model patch", based on a fusion strategy that integrates salience and sensitivity analysis. Subsequently, a compensation mechanism is introduced to "decorate the patch", enhancing the model's performance on both target and original tasks. Additionally, our method is adaptable to multi-task scenarios. Through extensive experiments on InstructBLIP and LLaVA-1.5 in both image captioning and visual question answering tasks, our approach demonstrates significant task adaptability while preserving inherent pre-trained capabilities.

翻译：灾难性遗忘在微调多模态大语言模型（MLLMs）时成为一个关键挑战，即在提升未见任务性能的同时，往往会导致原始任务性能显著下降。本文对MLLMs中的灾难性遗忘进行了全面分析，并提出了一种名为“模型裁缝”（Model Tailor）的训练后调整方法。我们的方法主要保留预训练参数，同时替换少量（≤10%）的微调参数，在原始任务上保持约99%的预训练效果，在新任务上达到与标准微调相当的约97%性能。具体而言，我们基于融合显著性和敏感性分析的策略，推导出稀疏掩码以识别“模型补丁”。随后，引入补偿机制来“装饰补丁”，增强模型在目标任务和原始任务上的表现。此外，我们的方法可适应多任务场景。通过在图像描述和视觉问答任务上对InstructBLIP和LLaVA-1.5进行广泛实验，我们的方法在保持固有预训练能力的同时，展现出显著的任务适应性。