Fine-tuning large-scale pre-trained models via transfer learning is an emerging important paradigm for a wide range of downstream tasks, with performance heavily reliant on extensive data. Federated learning (FL), as a distributed framework, provides a secure solution to train models on local datasets while safeguarding raw sensitive data. However, FL networks encounter high communication costs due to the massive parameters of large-scale pre-trained models, necessitating parameter-efficient methods. Notably, parameter efficient fine tuning, such as Low-Rank Adaptation (LoRA), has shown remarkable success in fine-tuning pre-trained models. However, prior research indicates that the fixed parameter budget may be prone to the overfitting or slower convergence. To address this challenge, we propose a Simulated Annealing-based Federated Learning with LoRA tuning (SA-FedLoRA) approach by reducing trainable parameters. Specifically, SA-FedLoRA comprises two stages: initiating and annealing. (1) In the initiating stage, we implement a parameter regularization approach during the early rounds of aggregation, aiming to mitigate client drift and accelerate the convergence for the subsequent tuning. (2) In the annealing stage, we allocate higher parameter budget during the early 'heating' phase and then gradually shrink the budget until the 'cooling' phase. This strategy not only facilitates convergence to the global optimum but also reduces communication costs. Experimental results demonstrate that SA-FedLoRA is an efficient FL, achieving superior performance to FedAvg and significantly reducing communication parameters by up to 93.62%.

翻译：通过迁移学习对大规模预训练模型进行微调已成为广泛下游任务中新兴的重要范式，其性能高度依赖于充足的数据。联邦学习作为一种分布式框架，在保护原始敏感数据的同时，为在本地数据集上训练模型提供了安全解决方案。然而，由于大规模预训练模型参数量庞大，联邦学习网络面临高通信成本的问题，亟需参数高效方法。值得注意的是，低秩适应（LoRA）等参数高效微调方法已在预训练模型微调中取得显著成功。但先前研究表明，固定参数预算可能导致过拟合或收敛速度较慢。为应对这一挑战，我们提出一种基于模拟退火的联邦学习与LoRA调优方法（SA-FedLoRA），通过减少可训练参数实现高效学习。具体而言，SA-FedLoRA包含两个阶段：初始阶段和退火阶段。（1）在初始阶段，我们于早期聚合轮次中实施参数正则化方法，旨在缓解客户端漂移并加速后续调优的收敛。（2）在退火阶段，我们先在早期"加热"阶段分配较高参数预算，随后逐步缩减预算直至"冷却"阶段。该策略不仅有助于收敛至全局最优解，还能降低通信成本。实验结果表明，SA-FedLoRA是一种高效的联邦学习方法，其性能优于FedAvg，且通信参数最高减少93.62%。