With the development of edge networks and mobile computing, the need to serve heterogeneous data sources at the network edge requires the design of new distributed machine learning mechanisms. As a prevalent approach, Federated Learning (FL) employs parameter-sharing and gradient-averaging between clients and a server. Despite its many favorable qualities, such as convergence and data-privacy guarantees, it is well-known that classic FL fails to address the challenge of data heterogeneity and computation heterogeneity across clients. Most existing works that aim to accommodate such sources of heterogeneity stay within the FL operation paradigm, with modifications to overcome the negative effect of heterogeneous data. In this work, as an alternative paradigm, we propose a Multi-Task Split Learning (MTSL) framework, which combines the advantages of Split Learning (SL) with the flexibility of distributed network architectures. In contrast to the FL counterpart, in this paradigm, heterogeneity is not an obstacle to overcome, but a useful property to take advantage of. As such, this work aims to introduce a new architecture and methodology to perform multi-task learning for heterogeneous data sources efficiently, with the hope of encouraging the community to further explore the potential advantages we reveal. To support this promise, we first show through theoretical analysis that MTSL can achieve fast convergence by tuning the learning rate of the server and clients. Then, we compare the performance of MTSL with existing multi-task FL methods numerically on several image classification datasets to show that MTSL has advantages over FL in training speed, communication cost, and robustness to heterogeneous data.

翻译：随着边缘网络和移动计算的发展，服务网络边缘异构数据源的需求催生了新型分布式机器学习机制的设计。联邦学习（FL）作为一种主流方法，在客户端与服务器之间采用参数共享与梯度平均。尽管具备收敛性与数据隐私保障等诸多优良特性，但经典FL众所周知地无法应对客户端间数据异构性与计算异构性的挑战。现有大多数旨在适应此类异构性的研究仍局限于FL操作范式，仅通过改进来克服异构数据的负面影响。本工作提出一种替代范式——多任务分割学习（MTSL）框架，该框架融合了分割学习（SL）的优势与分布式网络架构的灵活性。与FL范式不同，在此范式中异构性并非需要克服的障碍，而是可供利用的有益特性。因此，本工作旨在引入一种新的架构与方法论，以高效执行面向异构数据源的多任务学习，期望推动学界进一步探索我们所揭示的潜在优势。为验证这一前景，我们首先通过理论分析表明MTSL可通过调节服务器与客户端的学习率实现快速收敛。随后，我们在多个图像分类数据集上数值化比较了MTSL与现有多任务FL方法的性能，证明MTSL在训练速度、通信成本及对异构数据的鲁棒性方面均优于FL。