Parameter-efficient fine-tuning (PEFT) methods seek to adapt large models via updates to a small number of weights. However, much prior interpretability work has shown that representations encode rich semantic information, suggesting that editing representations might be a more powerful alternative. Here, we pursue this hypothesis by developing a family of $\textbf{Representation Finetuning (ReFT)}$ methods. ReFT methods operate on a frozen base model and learn task-specific interventions on hidden representations. We define a strong instance of the ReFT family, Low-rank Linear Subspace ReFT (LoReFT). LoReFT is a drop-in replacement for existing PEFTs and learns interventions that are 10x-50x more parameter-efficient than prior state-of-the-art PEFTs. We showcase LoReFT on eight commonsense reasoning tasks, four arithmetic reasoning tasks, Alpaca-Eval v1.0, and GLUE. In all these evaluations, LoReFT delivers the best balance of efficiency and performance, and almost always outperforms state-of-the-art PEFTs. We release a generic ReFT training library publicly at https://github.com/stanfordnlp/pyreft.

翻译：参数高效微调（PEFT）方法旨在通过更新少量权重来适应大模型。然而，大量先前的可解释性研究表明，表示（representations）编码了丰富的语义信息，这意味着编辑表示或许是一种更强大的替代方案。为此，我们通过开发一系列$\textbf{表示微调（Representation Finetuning，ReFT）}$方法来验证这一假设。ReFT方法基于冻结的基座模型运行，并在隐藏表示上学习任务特定的干预操作。我们定义了ReFT家族中的一个强效实例——低秩线性子空间ReFT（LoReFT）。LoReFT可直接替代现有PEFT方法，其学习的干预操作在参数效率上比先前最先进的PEFT方法高出10-50倍。我们在八个常识推理任务、四个算术推理任务、Alpaca-Eval v1.0以及GLUE基准上展示了LoReFT的性能。在所有评估中，LoReFT均在效率与性能之间实现了最佳平衡，且几乎始终优于最先进的PEFT方法。我们已在https://github.com/stanfordnlp/pyreft 公开了通用的ReFT训练库。