Fine-tuning large-scale Transformers has led to the explosion of many AI applications across Natural Language Processing and Computer Vision tasks. However, fine-tuning all pre-trained model parameters becomes impractical as the model size and number of tasks increase. Parameter-efficient transfer learning (PETL) methods aim to address these challenges. While effective in reducing the number of trainable parameters, PETL methods still require significant energy and computational resources to fine-tune. In this paper, we introduce \textbf{RE}current \textbf{AD}aption (READ) -- a lightweight and memory-efficient fine-tuning method -- to overcome the limitations of the current PETL approaches. Specifically, READ inserts a small RNN network alongside the backbone model so that the model does not have to back-propagate through the large backbone network. Through comprehensive empirical evaluation of the GLUE benchmark, we demonstrate READ can achieve a $56\%$ reduction in the training memory consumption and an $84\%$ reduction in the GPU energy usage while retraining high model quality compared to full-tuning. Additionally, the model size of READ does not grow with the backbone model size, making it a highly scalable solution for fine-tuning large Transformers.

翻译：微调大规模Transformer已推动了自然语言处理和计算机视觉任务中众多AI应用的爆发式发展。然而，随着模型规模和任务数量的增加，微调所有预训练模型参数变得不切实际。参数高效迁移学习（PETL）方法旨在解决这些挑战。尽管在减少可训练参数数量方面效果显著，PETL方法在微调过程中仍需要大量能源和计算资源。本文提出一种轻量级且内存高效的微调方法——循环适应（READ），以克服当前PETL方法的局限性。具体而言，READ在骨干模型旁插入一个小型RNN网络，从而避免模型通过大型骨干网络进行反向传播。通过对GLUE基准的综合实验评估，我们证明，与全量微调相比，READ在保持高模型质量的同时，可将训练内存消耗降低56%，GPU能耗降低84%。此外，READ的模型规模不随骨干模型规模增长，使其成为微调大型Transformer的高度可扩展方案。