Large-scale neural networks possess considerable expressive power. They are well-suited for complex learning tasks in industrial applications. However, large-scale models pose significant challenges for training under the current Federated Learning (FL) paradigm. Existing approaches for efficient FL training often leverage model parameter dropout. However, manipulating individual model parameters is not only inefficient in meaningfully reducing the communication overhead when training large-scale FL models, but may also be detrimental to the scaling efforts and model performance as shown by recent research. To address these issues, we propose the Federated Opportunistic Block Dropout (FedOBD) approach. The key novelty is that it decomposes large-scale models into semantic blocks so that FL participants can opportunistically upload quantized blocks, which are deemed to be significant towards training the model, to the FL server for aggregation. Extensive experiments evaluating FedOBD against four state-of-the-art approaches based on multiple real-world datasets show that it reduces the overall communication overhead by more than 88% compared to the best performing baseline approach, while achieving the highest test accuracy. To the best of our knowledge, FedOBD is the first approach to perform dropout on FL models at the block level rather than at the individual parameter level.

翻译：摘要：大规模神经网络具有强大的表达能力，特别适合工业应用中复杂的机器学习任务。然而，在当前联邦学习（FL）范式下，大规模模型的训练面临重大挑战。现有高效联邦学习方法常采用模型参数丢弃策略，但操纵单个模型参数不仅难以有效降低大规模FL模型训练中的通信开销，近期研究更表明这可能损害模型的可扩展性与性能。为解决这些问题，我们提出联邦机会性块丢弃（FedOBD）方法。其关键创新在于将大规模模型分解为语义模块，使FL参与者能够机会性地将量化后的、对模型训练具有重要意义的模块上传至FL服务器进行聚合。基于多个真实数据集，将FedOBD与四种最先进方法进行对比实验，结果表明：相比最优基线方法，该方法在实现最高测试精度的同时，整体通信开销降低超过88%。据我们所知，FedOBD是首个在块级别而非单个参数级别对FL模型执行丢弃操作的方法。