Statistical data heterogeneity is a significant barrier to convergence in federated learning (FL). While prior work has advanced heterogeneous FL through better optimization objectives, these methods fall short when there is extreme data heterogeneity among collaborating participants. We hypothesize that convergence under extreme data heterogeneity is primarily hindered due to the aggregation of conflicting updates from the participants in the initial collaboration rounds. To overcome this problem, we propose a warmup phase where each participant learns a personalized mask and updates only a subnetwork of the full model. This personalized warmup allows the participants to focus initially on learning specific subnetworks tailored to the heterogeneity of their data. After the warmup phase, the participants revert to standard federated optimization, where all parameters are communicated. We empirically demonstrate that the proposed personalized warmup via subnetworks (FedPeWS) approach improves accuracy and convergence speed over standard federated optimization methods.

翻译：统计数据的异构性是联邦学习（FL）收敛的一个主要障碍。尽管先前的研究通过改进优化目标推动了异构联邦学习的发展，但当协作参与者之间存在极端数据异构性时，这些方法仍显不足。我们假设，极端数据异构性下的收敛主要受阻于初始协作轮次中参与者之间冲突性更新的聚合。为克服此问题，我们提出一个预热阶段，其中每个参与者学习一个个性化掩码，并仅更新完整模型的一个子网络。这种个性化预热使参与者能够首先专注于学习针对其数据异构性定制的特定子网络。预热阶段结束后，参与者恢复至标准的联邦优化过程，此时所有参数均进行通信。我们通过实验证明，所提出的通过子网络进行个性化预热（FedPeWS）的方法，相较于标准联邦优化方法，在准确性和收敛速度上均有所提升。