Federated Learning (FL) requires frequent exchange of model parameters, which leads to long communication delay, especially when the network environments of clients vary greatly. Moreover, the parameter server needs to wait for the slowest client (i.e., straggler, which may have the largest model size, lowest computing capability or worst network condition) to upload parameters, which may significantly degrade the communication efficiency. Commonly-used client selection methods such as partial client selection would lead to the waste of computing resources and weaken the generalization of the global model. To tackle this problem, along a different line, in this paper, we advocate the approach of model parameter dropout instead of client selection, and accordingly propose a novel framework of Federated learning scheme with Differential parameter Dropout (FedDD). FedDD consists of two key modules: dropout rate allocation and uploaded parameter selection, which will optimize the model parameter uploading ratios tailored to different clients' heterogeneous conditions and also select the proper set of important model parameters for uploading subject to clients' dropout rate constraints. Specifically, the dropout rate allocation is formulated as a convex optimization problem, taking system heterogeneity, data heterogeneity, and model heterogeneity among clients into consideration. The uploaded parameter selection strategy prioritizes on eliciting important parameters for uploading to speedup convergence. Furthermore, we theoretically analyze the convergence of the proposed FedDD scheme. Extensive performance evaluations demonstrate that the proposed FedDD scheme can achieve outstanding performances in both communication efficiency and model convergence, and also possesses a strong generalization capability to data of rare classes.

翻译：摘要：联邦学习需要频繁交换模型参数，这会导致较长的通信延迟，尤其是在客户端网络环境差异显著的情况下。此外，参数服务器需要等待最慢的客户端（即掉队者，其可能具有最大的模型规模、最低的计算能力或最差的网络条件）上传参数，这可能会显著降低通信效率。常用的客户端选择方法（如部分客户端选择）会导致计算资源浪费并削弱全局模型的泛化能力。为解决这一问题，本文另辟蹊径，提出采用模型参数丢弃而非客户端选择的方法，并据此设计了一种新颖的差分参数丢弃联邦学习框架（FedDD）。FedDD包含两个关键模块：丢弃率分配与上传参数选择。前者针对不同客户端的异质条件优化模型参数上传比例，后者则根据客户端的丢弃率约束，选择合适的重要模型参数子集进行上传。具体而言，丢弃率分配被建模为一个凸优化问题，综合考虑了客户端之间的系统异质性、数据异质性与模型异质性；上传参数选择策略则优先提取重要参数以加速收敛。此外，我们从理论上分析了所提FedDD方案的收敛性。大量性能评估表明，FedDD方案在通信效率和模型收敛方面均能实现卓越性能，并且对稀有类别数据具有强大的泛化能力。