Fine-tuning large pre-trained models has become the de facto strategy for developing both task-specific and general-purpose machine learning systems, including developing models that are safe to deploy. Despite its clear importance, there has been minimal work that explains how fine-tuning alters the underlying capabilities learned by a model during pretraining: does fine-tuning yield entirely novel capabilities or does it just modulate existing ones? We address this question empirically in synthetic, controlled settings where we can use mechanistic interpretability tools (e.g., network pruning and probing) to understand how the model's underlying capabilities are changing. We perform an extensive analysis of the effects of fine-tuning in these settings, and show that: (i) fine-tuning rarely alters the underlying model capabilities; (ii) a minimal transformation, which we call a 'wrapper', is typically learned on top of the underlying model capabilities, creating the illusion that they have been modified; and (iii) further fine-tuning on a task where such hidden capabilities are relevant leads to sample-efficient 'revival' of the capability, i.e., the model begins reusing these capability after only a few gradient steps. This indicates that practitioners can unintentionally remove a model's safety wrapper merely by fine-tuning it on a, e.g., superficially unrelated, downstream task. We additionally perform analysis on language models trained on the TinyStories dataset to support our claims in a more realistic setup.

翻译：微调大规模预训练模型已成为开发特定任务和通用机器学习系统（包括安全部署模型）的事实标准策略。尽管其重要性显而易见，但关于微调如何改变模型在预训练期间学习到的基础能力的研究却极为有限：微调是产生全新的能力，还是仅对现有能力进行调节？我们在合成、受控设置中通过实证研究解决这一问题，利用机制可解释性工具（如网络剪枝和探针分析）来理解模型基础能力的变化方式。我们在这些设置中对微调效果进行了广泛分析，结果表明：（i）微调极少改变模型的基础能力；（ii）通常在模型基础能力之上学习到一种称为“包装器”的最小变换，造成能力已被修改的假象；（iii）在相关任务（隐藏能力发挥作用）上进一步微调会以样本高效的方式“复兴”该能力，即模型仅需少量梯度更新步骤即可重新利用这些能力。这表明实践者只需在表面上无关的下游任务上进行微调，就可能无意中移除模型的安全包装器。此外，我们对基于TinyStories数据集训练的语言模型进行了分析，以在更真实的场景中支持我们的结论。