Federated learning (FL) is widely employed for collaborative training on decentralized data but faces challenges like data, system, and model heterogeneity. This prompted the emergency of model-heterogeneous personalized federated learning (MHPFL). However, concerns persist regarding data and model privacy, model performance, communication, and computational costs in current MHPFL methods. To tackle these concerns, we propose a novel model-heterogeneous personalized Federated learning algorithm (FedMoE) with the Mixture of Experts (MoE), renowned for enhancing large language models (LLMs). It assigns a shared homogeneous small feature extractor and a local gating network for each client's local heterogeneous large model. (1) During local training, the local heterogeneous model's feature extractor acts as a local expert for personalized feature (representation) extraction, while the shared homogeneous small feature extractor serves as a global expert for generalized feature extraction. The local gating network produces personalized weights for extracted representations from both experts on each data sample. The three models form a local heterogeneous MoE. The weighted mixed representation fuses global generalized and local personalized features and is processed by the local heterogeneous large model's header with personalized prediction information for output. The MoE and prediction header are updated synchronously. (2) The trained local homogeneous small feature extractors are sent to the server for cross-client information fusion via aggregation. Briefly, FedMoE first enhances local model personalization at a fine-grained data level while supporting model heterogeneity.

翻译：联邦学习（FL）广泛应用于去中心化数据的协同训练，但面临数据、系统与模型异构等挑战，催生了模型异构个性化联邦学习（MHPFL）的出现。然而，现有MHPFL方法在数据与模型隐私、模型性能、通信及计算开销方面仍存在隐患。为解决这些问题，我们提出一种基于混合专家（MoE）的新型模型异构个性化联邦学习算法（FedMoE），该算法因能增强大型语言模型（LLMs）而闻名。它为每个客户端的本地异构大模型配备共享的同构小型特征提取器和本地门控网络：（1）在本地训练阶段，本地异构模型的特征提取器作为本地专家执行个性化特征提取，而共享同构小型特征提取器作为全局专家进行泛化特征提取。本地门控网络针对每个数据样本为两个专家提取的表征生成个性化权重。三者构成本地异构MoE，加权混合表征融合了全局泛化特征与本地个性化特征，并交由包含个性化预测信息的本地异构大模型头部进行处理。MoE与预测头部同步更新；（2）训练后的本地同构小型特征提取器被发送至服务器，通过聚合实现跨客户端信息融合。简言之，FedMoE在支持模型异构性的同时，首先在细粒度数据层面增强了本地模型的个性化能力。