We introduce FedDCT, a novel distributed learning paradigm that enables the usage of large, high-performance CNNs on resource-limited edge devices. As opposed to traditional FL approaches, which require each client to train the full-size neural network independently during each training round, the proposed FedDCT allows a cluster of several clients to collaboratively train a large deep learning model by dividing it into an ensemble of several small sub-models and train them on multiple devices in parallel while maintaining privacy. In this collaborative training process, clients from the same cluster can also learn from each other, further improving their ensemble performance. In the aggregation stage, the server takes a weighted average of all the ensemble models trained by all the clusters. FedDCT reduces the memory requirements and allows low-end devices to participate in FL. We empirically conduct extensive experiments on standardized datasets, including CIFAR-10, CIFAR-100, and two real-world medical datasets HAM10000 and VAIPE. Experimental results show that FedDCT outperforms a set of current SOTA FL methods with interesting convergence behaviors. Furthermore, compared to other existing approaches, FedDCT achieves higher accuracy and substantially reduces the number of communication rounds (with $4-8$ times fewer memory requirements) to achieve the desired accuracy on the testing dataset without incurring any extra training cost on the server side.

翻译：我们提出了一种新型分布式学习范式 FedDCT，它能够使资源受限的边缘设备运行高性能的大型卷积神经网络。与每个客户端需在每轮训练中独立训练完整神经网络的传统联邦学习方法不同，FedDCT 通过将大规模深度学习模型分解为若干子模型集成，并允许同一集群内的多个客户端在保护隐私的前提下并行协作训练这些子模型。在此协作训练过程中，同集群的客户端可相互学习，进一步提升集成模型性能。在聚合阶段，服务器对所有集群训练的集成模型进行加权平均。FedDCT 降低了内存需求，使低端设备得以参与联邦学习。我们在标准化数据集（CIFAR-10、CIFAR-100）及两个真实医学数据集 HAM10000 和 VAIPE 上开展了大量实验。结果表明，FedDCT 在收敛行为方面优于当前一系列最先进的联邦学习方法。此外，与现有方法相比，FedDCT 可在不增加服务器端训练成本的情况下，以 4-8 倍较低的内存需求实现更高精度，并大幅减少通信轮次。