Parameter-efficient finetuning (PEFT) has become the standard approach for adapting large language models, yet evaluations largely emphasize downstream accuracy while overlooking the retention of pretrained capabilities. We argue that PEFT should be assessed through the stability-plasticity dilemma: the trade-off between target-task adaptation and resistance to forgetting. We introduce PEFT-Arena, a benchmark that jointly measures downstream performance and general capability retention. Across methods, we find distinct stability-plasticity profiles; under comparable parameter budgets, orthogonal finetuning achieves the most favorable Pareto frontier. To explain these differences, we analyze PEFT updates from two geometric perspectives. In weight space, spectral analysis reveals how parameterizations interact with the pretrained singular-value structure. In activation space, retention metrics show whether finetuning preserves or distorts general-capability representations, with forgetting linked to non-isometric representation distortion. Finally, an analysis shows that final SFT checkpoints often overshoot a better target-retention operating point. Inspired by this, we present case studies of a post-hoc improvement with path-wise rewinding.

翻译：参数高效微调（PEFT）已成为适配大语言模型的标准方法，然而现有评估主要关注下游任务准确率，忽略了预训练能力的保持。我们认为应从稳定性-可塑性困境来评估PEFT：即目标任务适配与抗遗忘能力之间的权衡。我们提出PEFT-Arena基准，可联合衡量下游性能与通用能力保持。跨方法比较发现，不同方法具有独特的稳定性-可塑性特征；在可比参数预算下，正交微调实现了最优的帕累托前沿。为解释这些差异，我们从两个几何视角分析PEFT更新：在权重空间，频谱分析揭示了参数化方式如何与预训练奇异值结构相互作用；在激活空间，保持指标显示微调是保留还是扭曲通用能力表征，其中遗忘与非等距表征扭曲相关。最后，分析表明最终SFT检查点常偏离更优的目标-保持运行点。受此启发，我们提出基于路径回退的事后改进案例研究。