Decentralized learning (DL) leverages edge devices for collaborative model training while avoiding coordination by a central server. Due to privacy concerns, DL has become an attractive alternative to centralized learning schemes since training data never leaves the device. In a round of DL, all nodes participate in model training and exchange their model with some other nodes. Performing DL in large-scale heterogeneous networks results in high communication costs and prolonged round durations due to slow nodes, effectively inflating the total training time. Furthermore, current DL algorithms also assume all nodes are available for training and aggregation at all times, diminishing the practicality of DL. This paper presents Plexus, an efficient, scalable, and practical DL system. Plexus (1) avoids network-wide participation by introducing a decentralized peer sampler that selects small subsets of available nodes that train the model each round and, (2) aggregates the trained models produced by nodes every round. Plexus is designed to handle joining and leaving nodes (churn). We extensively evaluate Plexus by incorporating realistic traces for compute speed, pairwise latency, network capacity, and availability of edge devices in our experiments. Our experiments on four common learning tasks empirically show that Plexus reduces time-to-accuracy by 1.2-8.3x, communication volume by 2.4-15.3x and training resources needed for convergence by 6.4-370x compared to baseline DL algorithms.

翻译：去中心化学习利用边缘设备进行协作模型训练，同时避免中央服务器的协调。由于隐私问题，训练数据始终保留在设备端，使得去中心化学习成为集中式学习方案颇具吸引力的替代方案。在每轮去中心化学习中，所有节点参与模型训练并与部分其他节点交换模型。在大规模异构网络中实施去中心化学习会导致高通信成本和因慢节点造成的长时间轮次延迟，从而显著增加总训练时间。此外，现有去中心化学习算法还假设所有节点始终可用于训练和聚合，这降低了其实际可行性。本文提出Plexus——一种高效、可扩展且实用的去中心化学习系统。Plexus通过以下设计实现：(1) 引入去中心化对等采样器，每轮选择小部分可用节点子集参与模型训练，避免全网参与；(2) 每轮聚合各节点产生的训练模型。该系统特别针对节点动态加入与退出（节点波动）场景进行优化。我们通过集成边缘设备的实际轨迹数据（涵盖计算速度、点对点延迟、网络容量和可用性）对Plexus进行全面评估。在四项常见学习任务上的实验结果表明，与传统去中心化学习算法相比，Plexus可将达到目标精度所需时间缩短1.2-8.3倍，通信量降低2.4-15.3倍，收敛所需训练资源减少6.4-370倍。