Federated learning (FedL) has emerged as a popular technique for distributing model training over a set of wireless devices, via iterative local updates (at devices) and global aggregations (at the server). In this paper, we develop parallel successive learning (PSL), which expands the FedL architecture along three dimensions: (i) Network, allowing decentralized cooperation among the devices via device-to-device (D2D) communications. (ii) Heterogeneity, interpreted at three levels: (ii-a) Learning: PSL considers heterogeneous number of stochastic gradient descent iterations with different mini-batch sizes at the devices; (ii-b) Data: PSL presumes a dynamic environment with data arrival and departure, where the distributions of local datasets evolve over time, captured via a new metric for model/concept drift. (ii-c) Device: PSL considers devices with different computation and communication capabilities. (iii) Proximity, where devices have different distances to each other and the access point. PSL considers the realistic scenario where global aggregations are conducted with idle times in-between them for resource efficiency improvements, and incorporates data dispersion and model dispersion with local model condensation into FedL. Our analysis sheds light on the notion of cold vs. warmed up models, and model inertia in distributed machine learning. We then propose network-aware dynamic model tracking to optimize the model learning vs. resource efficiency tradeoff, which we show is an NP-hard signomial programming problem. We finally solve this problem through proposing a general optimization solver. Our numerical results reveal new findings on the interdependencies between the idle times in-between the global aggregations, model/concept drift, and D2D cooperation configuration.

翻译：联邦学习（FedL）已成为一种在无线设备集上通过迭代局部更新（设备端）和全局聚合（服务器端）进行分布式模型训练的流行技术。本文提出并行连续学习（PSL），该架构沿三个维度扩展了FedL：(i) 网络：允许设备通过设备到设备（D2D）通信实现去中心化协作。(ii) 异构性：在三个层面诠释：(ii-a) 学习：PSL考虑设备端不同小批量尺寸下的异构随机梯度下降迭代次数；(ii-b) 数据：PSL假设数据到达与离开的动态环境，本地数据集分布随时间演变，通过新的模型/概念漂移度量进行刻画；(ii-c) 设备：PSL考虑计算与通信能力各异的设备。(iii) 邻近性：设备间及设备与接入点间存在不同距离。PSL考虑实际场景中全局聚合间存在空闲时间以提升资源效率，并将数据分散、模型分散与局部模型压缩融入FedL。我们的分析揭示了分布式机器学习中冷/热启动模型及模型惯性的概念内涵。进而提出网络感知动态模型追踪方法以优化模型学习与资源效率的权衡，该问题被证明是NP难的符号几何规划问题。最终通过提出通用优化求解器解决该问题。数值结果揭示了全局聚合间空闲时间、模型/概念漂移与D2D协作配置之间的相互依赖关系新发现。