Adapting a pretrained language model to a new task often hurts the general capabilities it already had, a problem known as catastrophic forgetting. Sparse Memory Finetuning (SMF) tries to avoid this by adding key-value memory layers to the model and, on each training step, updating only the small set of memory rows that the current batch reads most heavily. We re-implement SMF on Qwen-2.5-0.5B-Instruct and compare it with LoRA and full finetuning on MedMCQA, a 4-choice medical exam task, using WikiText perplexity and TriviaQA accuracy as forgetting probes. SMF improves MedMCQA by 2.5 percentage points while keeping both forgetting probes within roughly 1 point of the base model, whereas LoRA and full finetuning achieve larger gains but with clear drift on both. We also compare two row-selection rules (KL-divergence and TF-IDF), which balance the two forgetting metrics differently.

翻译：将预训练语言模型适配至新任务时，往往会损害其已具备的通用能力，这种现象被称为灾难性遗忘。稀疏记忆微调（SMF）通过向模型添加键值记忆层，并在每步训练中仅更新当前批次读取最频繁的小部分记忆行来规避该问题。我们在Qwen-2.5-0.5B-Instruct上复现SMF，将其与LoRA及全参数微调在MedMCQA（四选一医学考试任务）中进行对比，并采用WikiText困惑度与TriviaQA准确率作为遗忘探测指标。实验表明：SMF使MedMCQA准确率提升2.5个百分点，同时将两项遗忘探测指标控制在与基模型相差约1个百分点范围内；而LoRA与全参数微调虽获得更大增益，但在两个探测指标上均出现明显偏移。此外，我们比较了两种行选择规则（KL散度与TF-IDF），两者对两类遗忘指标的权衡效果存在差异。