Transformer-based large language models exhibit in-context learning, enabling adaptation to downstream tasks via few-shot prompting with demonstrations. In practice, such models are often fine-tuned to improve zero-shot performance on downstream tasks, allowing them to solve tasks without examples and thereby reducing inference costs. However, fine-tuning can degrade in-context learning, limiting the performance of fine-tuned models on tasks not seen during fine-tuning. Using linear attention models, we provide a theoretical analysis that characterizes how fine-tuning objectives modify attention parameters and identifies conditions under which this leads to degraded few-shot performance. We show that fine-tuning all attention parameters can harm in-context learning, whereas restricting updates to the value matrix improves zero-shot performance while preserving in-context learning. We further show that incorporating an auxiliary few-shot loss enhances in-context learning primarily on the target task, at the expense of degraded in-context learning ability on tasks not seen during fine-tuning. We empirically validate our theoretical results.

翻译：基于Transformer的大型语言模型展现出上下文学习能力，使其能够通过包含少量示例的提示来适应下游任务。在实践中，此类模型通常经过微调以提升其在下游任务上的零样本性能，从而使其能够在无需示例的情况下完成任务，进而降低推理成本。然而，微调可能会损害上下文学习能力，限制微调模型在微调期间未见任务上的表现。我们利用线性注意力模型，提供了一个理论分析，刻画了微调目标如何修改注意力参数，并识别了导致少样本性能下降的条件。我们证明，微调所有注意力参数可能损害上下文学习，而将更新限制在值矩阵上则能在保持上下文学习的同时改善零样本性能。我们进一步表明，引入辅助的少样本损失主要能增强模型在目标任务上的上下文学习，但会以牺牲在微调期间未见任务上的上下文学习能力为代价。我们的理论结果得到了实验验证。