We investigate parameter-efficient fine-tuning (PEFT) methods that can provide good accuracy under limited computational and memory budgets in the context of large language models (LLMs). We present a new PEFT method called Robust Adaptation (RoSA) inspired by robust principal component analysis (PCA) that jointly trains $\textit{low-rank}$ and $\textit{highly-sparse}$ components on top of a set of fixed pretrained weights to efficiently approximate the performance of a full-fine-tuning (FFT) solution. Across a series of challenging generative tasks such as grade-school math and SQL query generation, which require fine-tuning for good performance, we show that RoSA outperforms both LoRA and pure sparse fine-tuning, at the same parameter budget. We provide system support for RoSA to complement the training algorithm, specifically in the form of sparse GPU kernels which enable memory- and computationally-efficient training. Our code will be made available at $\href{https://github.com/IST-DASLab/RoSA}{\text{our github page}}$.

翻译：我们研究了在大型语言模型（LLM）计算和内存预算有限的背景下，能够提供良好精度的参数高效微调（PEFT）方法。受鲁棒主成分分析（PCA）启发，我们提出了一种名为鲁棒适应（RoSA）的新型PEFT方法，该方法在固定预训练权重之上联合训练$\textit{低秩}$和$\textit{高稀疏}$组件，以高效逼近全微调（FFT）解决方案的性能。在一系列具有挑战性的生成任务（如小学数学和SQL查询生成，需要微调以获得良好性能）中，我们表明，在相同参数预算下，RoSA的表现优于LoRA和纯稀疏微调。我们为RoSA提供了系统支持以补充训练算法，具体形式为稀疏GPU内核，可实现内存和计算高效训练。我们的代码将在$\href{https://github.com/IST-DASLab/RoSA}{\text{我们的GitHub页面}}$上提供。