Optimizers play an important role in both pretraining and finetuning stages when training large language models (LLMs). In this paper, we present an observation that full finetuning with the same optimizer as in pretraining achieves a better learning-forgetting tradeoff, i.e., forgetting less while achieving the same or better performance on the new task, than other optimizers and, possibly surprisingly, LoRA, during the supervised finetuning (SFT) stage. We term this phenomenon optimizer-model consistency. To better understand it, through controlled experiments and theoretical analysis, we show that: 1) optimizers can shape the models by having regularization effects on the activations, leading to different landscapes around the pretrained checkpoints; 2) in response to this regularization effect, the weight update in SFT should follow some specific structures to lower forgetting of the knowledge learned in pretraining, which can be obtained by using the same optimizer. Moreover, we specifically compare Muon and AdamW when they are employed throughout the pretraining and SFT stages and find that Muon performs worse when finetuned for reasoning tasks. With a synthetic language modeling experiment, we demonstrate that this can come from Muon's strong tendency towards rote memorization, which may hurt pattern acquisition with a small amount of data, as for SFT.

翻译：优化器在训练大型语言模型（LLMs）的预训练和微调阶段均扮演着重要角色。本文发现，在监督微调（SFT）阶段，使用与预训练相同优化器进行全量微调相比其他优化器（可能令人惊讶的是，还包括LoRA）能实现更优的学习-遗忘权衡，即在新任务上取得相同或更优性能的同时遗忘更少。我们将此现象称为优化器-模型一致性。为深入理解该现象，通过受控实验和理论分析，我们揭示：1）优化器通过对激活值产生正则化效应来塑造模型，导致预训练检查点周围形成不同损失景观；2）针对这种正则化效应，SFT阶段的权重更新应遵循特定结构以降低对预训练知识的遗忘，而使用相同优化器即可获得该结构。此外，我们专门比较了Muon和AdamW在预训练和SFT全阶段使用时的表现，发现Muon在推理任务微调时表现更差。通过合成语言建模实验，我们证明这可能源于Muon强烈的机械记忆倾向，该倾向在SFT等数据量较少的场景中可能损害模式习得能力。