Federated learning (FL) facilitates edge devices to cooperatively train a global shared model while maintaining the training data locally and privately. However, a common but impractical assumption in FL is that the participating edge devices possess the same required resources and share identical global model architecture. In this study, we propose a novel FL method called Federated Intermediate Layers Learning (FedIN), supporting heterogeneous models without utilizing any public dataset. The training models in FedIN are divided into three parts, including an extractor, the intermediate layers, and a classifier. The model architectures of the extractor and classifier are the same in all devices to maintain the consistency of the intermediate layer features, while the architectures of the intermediate layers can vary for heterogeneous devices according to their resource capacities. To exploit the knowledge from features, we propose IN training, training the intermediate layers in line with the features from other clients. Additionally, we formulate and solve a convex optimization problem to mitigate the gradient divergence problem induced by the conflicts between the IN training and the local training. The experiment results show that FedIN achieves the best performance in the heterogeneous model environment compared with the state-of-the-art algorithms. Furthermore, our ablation study demonstrates the effectiveness of IN training and the solution to the convex optimization problem.

翻译：联邦学习（FL）使得边缘设备能够在保持训练数据本地化和私密性的同时，协同训练一个全局共享模型。然而，联邦学习中一个常见但不切实际的假设是，参与边缘设备拥有相同的所需资源并共享相同的全局模型架构。本研究提出一种名为联邦中间层学习（FedIN）的新型联邦学习方法，该方法在不使用任何公共数据集的情况下支持异构模型。FedIN中的训练模型分为三部分：提取器、中间层和分类器。所有设备中提取器和分类器的模型架构相同，以保持中间层特征的一致性；而中间层的架构可根据异构设备的资源能力进行变化。为利用特征中的知识，我们提出IN训练方法，使中间层的训练与其他客户端产生的特征保持一致。此外，我们构建并求解了一个凸优化问题，以缓解由IN训练与本地训练之间冲突引起的梯度差异问题。实验结果表明，与现有最优算法相比，FedIN在异构模型环境中取得了最佳性能。同时，消融实验证明了IN训练及凸优化问题求解的有效性。