Cross-device Federated Learning (FL) faces significant challenges where low-end clients that could potentially make unique contributions are excluded from training large models due to their resource bottlenecks. Recent research efforts have focused on model-heterogeneous FL, by extracting reduced-size models from the global model and applying them to local clients accordingly. Despite the empirical success, general theoretical guarantees of convergence on this method remain an open question. In this paper, we present a unifying framework for heterogeneous FL algorithms with online model extraction and provide a general convergence analysis. In particular, we prove that under certain sufficient conditions and for both IID and non-IID data, these algorithms converge to a stationary point of standard FL for general smooth cost functions. Moreover, we illuminate two key factors impacting its convergence: model-extraction noise and minimum coverage index, advocating a joint design of local model extraction for efficient heterogeneous FL.

翻译：跨设备联邦学习面临重大挑战：可能做出独特贡献的低端客户端因其资源瓶颈而被排除在大型模型训练之外。近期研究聚焦于模型异构联邦学习，通过从全局模型中提取缩减模型并分别应用于本地客户端。尽管取得了经验成功，但该方法收敛性的通用理论保证仍是未解难题。本文提出了一个集成在线模型提取的异构联邦学习算法统一框架，并提供了通用收敛性分析。具体而言，我们证明在特定充分条件下，对于独立同分布与非独立同分布数据，这些算法都能收敛到标准联邦学习关于一般光滑代价函数的稳定点。此外，我们揭示了影响其收敛性的两个关键因素：模型提取噪声与最小覆盖指数，倡导为高效异构联邦学习联合设计本地模型提取方案。