Federated Learning (FL) requires frequent exchange of model parameters, which leads to long communication delay, especially when the network environments of clients vary greatly. Moreover, the parameter server needs to wait for the slowest client (i.e., straggler, which may have the largest model size, lowest computing capability or worst network condition) to upload parameters, which may significantly degrade the communication efficiency. Commonly-used client selection methods such as partial client selection would lead to the waste of computing resources and weaken the generalization of the global model. To tackle this problem, along a different line, in this paper, we advocate the approach of model parameter dropout instead of client selection, and accordingly propose a novel framework of Federated learning scheme with Differential parameter Dropout (FedDD). FedDD consists of two key modules: dropout rate allocation and uploaded parameter selection, which will optimize the model parameter uploading ratios tailored to different clients' heterogeneous conditions and also select the proper set of important model parameters for uploading subject to clients' dropout rate constraints. Specifically, the dropout rate allocation is formulated as a convex optimization problem, taking system heterogeneity, data heterogeneity, and model heterogeneity among clients into consideration. The uploaded parameter selection strategy prioritizes on eliciting important parameters for uploading to speedup convergence. Furthermore, we theoretically analyze the convergence of the proposed FedDD scheme. Extensive performance evaluations demonstrate that the proposed FedDD scheme can achieve outstanding performances in both communication efficiency and model convergence, and also possesses a strong generalization capability to data of rare classes.

翻译：联邦学习需频繁交换模型参数，这导致通信延迟较长，特别是在客户端网络环境差异显著时。此外，参数服务器需等待最慢客户端（即掉队者——可能拥有最大模型规模、最低计算能力或最差网络条件）上传参数，这会显著降低通信效率。常见的客户端选择方法（如部分客户端选择）会导致计算资源浪费并削弱全局模型的泛化能力。针对此问题，本文另辟蹊径，倡导采用模型参数丢弃而非客户端选择的方案，并据此提出一种基于分差参数丢弃的联邦学习新框架（FedDD）。FedDD包含两个核心模块：丢弃率分配与上传参数选择。前者针对不同客户的异构条件优化模型参数上传比例，后者在满足客户端丢弃率约束的前提下挑选重要模型参数子集进行上传。具体而言，丢弃率分配被建模为考虑客户端间系统异构性、数据异构性与模型异构性的凸优化问题；上传参数选择策略优先提取重要参数以加速收敛。进一步，我们从理论上分析了FedDD方案的收敛性。大量性能评估表明，FedDD方案在通信效率与模型收敛性方面均表现卓越，并具备对稀有类别数据的强泛化能力。