Federated learning (FL) has emerged as a promising paradigm for enabling the collaborative training of models without centralized access to the raw data on local devices. In the typical FL paradigm (e.g., FedAvg), model weights are sent to and from the server each round to participating clients. However, this can quickly put a massive communication burden on the system, especially if more capable models beyond very small MLPs are employed. Recently, the use of pre-trained models has been shown effective in federated learning optimization and improving convergence. This opens the door for new research questions. Can we adjust the weight-sharing paradigm in federated learning, leveraging strong and readily-available pre-trained models, to significantly reduce the communication burden while simultaneously achieving excellent performance? To this end, we investigate the use of parameter-efficient fine-tuning in federated learning. Specifically, we systemically evaluate the performance of several parameter-efficient fine-tuning methods across a variety of client stability, data distribution, and differential privacy settings. By only locally tuning and globally sharing a small portion of the model weights, significant reductions in the total communication overhead can be achieved while maintaining competitive performance in a wide range of federated learning scenarios, providing insight into a new paradigm for practical and effective federated systems.

翻译：联邦学习(FL)已成为一个大有希望的模式,有利于在不集中获取当地设备原始数据的情况下对模型进行合作培训,这种培训模式是一个很有希望的范例,在典型的FL模式(例如FedAvg)中,模型的权重被发送到服务器,每个回合都从服务器传送到参与客户,然而,这可以很快给系统带来巨大的通信负担,特别是如果使用非常小的MLP以外的更有能力的模型的话。最近,在联合学习优化和改善融合方面,使用预先培训的模型已经证明是有效的。这为新的研究问题打开了大门。我们能否调整联合学习中的权重分担模式,利用强大和现成的事先培训模式,以大幅度降低通信负担,同时同时实现优异的绩效?为此目的,我们调查在联合学习过程中使用参数效率微调的系统,具体来说,我们系统评估在各种客户稳定、数据分布和不同隐私环境下采用若干参数高效的微调方法的绩效。只有局部调整和全球分享少量的模型重量,才能大幅度削减通信总间接费用,同时保持具有竞争力的全新思维模式,同时提供全局有竞争力的全局。