A primary function of back-propagation is to compute both the gradient of hidden representations and parameters for optimization with gradient descent. Training large models requires high computational costs due to their vast parameter sizes. While Parameter-Efficient Fine-Tuning (PEFT) methods aim to train smaller auxiliary models to save computational space, they still present computational overheads, especially in Fine-Tuning as a Service (FTaaS) for numerous users. We introduce Collaborative Adaptation (ColA) with Gradient Learning (GL), a parameter-free, model-agnostic fine-tuning approach that decouples the computation of the gradient of hidden representations and parameters. In comparison to PEFT methods, ColA facilitates more cost-effective FTaaS by offloading the computation of the gradient to low-cost devices. We also provide a theoretical analysis of ColA and experimentally demonstrate that ColA can perform on par or better than existing PEFT methods on various benchmarks.

翻译：摘要：反向传播的核心功能之一是计算隐层表示的梯度及参数，以通过梯度下降进行优化。由于大模型参数量庞大，其训练过程需要高昂的计算成本。尽管参数高效微调（PEFT）方法旨在训练更小规模的辅助模型以节省计算空间，但这些方法仍会产生额外计算开销，尤其在面向多用户的微调即服务（FTaaS）场景中尤为突出。本文提出基于梯度学习（GL）的协同适应（ColA）方法，这是一种无参数、模型无关的微调方法，通过解耦隐层表示梯度与参数梯度的计算过程实现优化。相较于PEFT方法，ColA将梯度计算任务卸载至低成本设备，从而显著降低FTaaS服务成本。我们提供了ColA的理论分析，并通过实验证明，在多个基准测试中，ColA的性能可与现有PEFT方法持平甚至更优。