Cross-device Federated Learning (FL) faces significant challenges where low-end clients that could potentially make unique contributions are excluded from training large models due to their resource bottlenecks. Recent research efforts have focused on model-heterogeneous FL, by extracting reduced-size models from the global model and applying them to local clients accordingly. Despite the empirical success, general theoretical guarantees of convergence on this method remain an open question. This paper presents a unifying framework for heterogeneous FL algorithms with online model extraction and provides a general convergence analysis for the first time. In particular, we prove that under certain sufficient conditions and for both IID and non-IID data, these algorithms converge to a stationary point of standard FL for general smooth cost functions. Moreover, we introduce the concept of minimum coverage index, together with model reduction noise, which will determine the convergence of heterogeneous federated learning, and therefore we advocate for a holistic approach that considers both factors to enhance the efficiency of heterogeneous federated learning.

翻译：跨设备联邦学习面临重大挑战，低端客户端可能做出独特贡献，但由于资源瓶颈而被排除在大型模型训练之外。近期研究聚焦于模型异构联邦学习，通过从全局模型中提取缩减规模模型，并相应地应用于本地客户端。尽管实证成功，但该方法收敛性的通用理论保证仍是一个开放问题。本文首次提出了一个统一的框架，用于具有在线模型提取的异构联邦学习算法，并提供了通用收敛性分析。特别地，我们证明在一定的充分条件下，对于独立同分布和非独立同分布数据，这些算法收敛到标准联邦学习针对一般平滑代价函数的驻点。此外，我们引入了最小覆盖指数的概念，连同模型缩减噪声，这将决定异构联邦学习的收敛性，因此我们倡导一种综合考虑这两个因素的整体方法，以提升异构联邦学习的效率。