Federated learning (FL) is a promising approach in distributed learning keeping privacy. However, during the training pipeline of FL, slow or incapable clients (i.e., stragglers) slow down the total training time and degrade performance. System heterogeneity, including heterogeneous computing and network bandwidth, has been addressed to mitigate the impact of stragglers. Previous studies split models to tackle the issue, but with less degree-of-freedom in terms of model architecture. We propose nested federated learning (NeFL), a generalized framework that efficiently divides a model into submodels using both depthwise and widthwise scaling. NeFL is implemented by interpreting models as solving ordinary differential equations (ODEs) with adaptive step sizes. To address the inconsistency that arises when training multiple submodels with different architecture, we decouple a few parameters. NeFL enables resource-constrained clients to effectively join the FL pipeline and the model to be trained with a larger amount of data. Through a series of experiments, we demonstrate that NeFL leads to significant gains, especially for the worst-case submodel (e.g., 8.33 improvement on CIFAR-10). Furthermore, we demonstrate NeFL aligns with recent studies in FL.

翻译：联邦学习（FL）是一种在分布式学习中兼具隐私保护特性的有前景方法。然而，在FL训练流程中，缓慢或能力不足的客户端（即掉队者）会延长整体训练时间并降低性能。针对计算与网络带宽的系统异构性已被研究以缓解掉队者的影响。现有研究通过分割模型应对该问题，但模型架构的自由度受限。我们提出嵌套联邦学习（NeFL），这是一种通过深度缩放与宽度缩放将模型高效划分为子模型的通用框架。NeFL将模型解读为具有自适应步长的常微分方程（ODE）来实现。为解决训练不同架构子模型时产生的不一致性，我们解耦了部分参数。NeFL使资源受限客户端能够有效参与FL流程，并使模型能够基于更大量数据进行训练。通过系列实验，我们证明NeFL在极端子模型场景中取得显著性能提升（如CIFAR-10上提升8.33%）。此外，我们验证了NeFL与近期FL研究成果的一致性。