Federated Learning (FL) and Split Learning (SL) are two popular paradigms of distributed machine learning. By offloading the computation-intensive portions to the server, SL is promising for deep model training on resource-constrained devices, yet still lacking of rigorous convergence analysis. In this paper, we derive the convergence guarantees of Sequential SL (SSL, the vanilla case of SL that conducts the model training in sequence) for strongly/general/non-convex objectives on heterogeneous data. Notably, the derived guarantees suggest that SSL is better than Federated Averaging (FedAvg, the most popular algorithm in FL) on heterogeneous data. We validate the counterintuitive analysis result empirically on extremely heterogeneous data.

翻译：联邦学习（FL）和分裂学习（SL）是分布式机器学习的两种流行范式。通过将计算密集型部分卸载到服务器，SL在资源受限设备上的深度模型训练方面具有前景，但仍缺乏严格的收敛性分析。本文针对异质数据上的强凸/一般凸/非凸目标函数，推导了序列式分裂学习（SSL，即SL的基本情形，按顺序进行模型训练）的收敛性保证。值得注意的是，推导出的保证表明，SSL在异质数据上优于联邦平均（FedAvg，FL中最流行的算法）。我们在极端异质数据上通过实验验证了这一反直觉的分析结果。