While federated learning (FL) enables fine-tuning of large language models (LLMs) without compromising data privacy, the substantial size of an LLM renders on-device training impractical for resource-constrained clients, such as mobile devices. Thus, Mixture-of-Experts (MoE) models have emerged as a computation-efficient solution, which activates only a sparse subset of experts during model training to reduce computing burden without sacrificing performance. Though integrating MoE into FL fine-tuning holds significant potential, it still encounters three key challenges: i) selecting appropriate experts for clients remains challenging due to the lack of a reliable metric to measure each expert's impact on local fine-tuning performance, ii) the heterogeneous computing resources across clients severely hinder MoE-based LLM fine-tuning, as dynamic expert activations across diverse input samples can overwhelm resource-constrained devices, and iii) client-specific expert subsets and routing preference undermine global aggregation, where misaligned expert updates and inconsistent gating networks in troduce destructive interference. To address these challenges, we propose HFedMoE, a heterogeneous MoE-based FL fine-tuning framework that customizes a subset of experts to each client for computation-efficient LLM fine-tuning. Specifically, HFedMoE identifies the expert importance based on its contributions to fine-tuning performance, and then adaptively selects a subset of experts from an information bottleneck perspective to align with each client' s computing budget. A sparsity-aware model aggregation strategy is also designed to aggregate the actively fine-tuned experts and gating parameters with importance weighted contributions. Extensive experiments demonstrate that HFedMoE outperforms state-of-the-art benchmarks in training accuracy and convergence speed.

翻译：尽管联邦学习（FL）能够在不损害数据隐私的前提下对大型语言模型（LLM）进行微调，但LLM的巨大规模使得在资源受限的客户端（如移动设备）上进行设备端训练变得不切实际。因此，专家混合（MoE）模型作为一种计算高效的解决方案应运而生，它仅在模型训练期间激活稀疏的专家子集，从而在不牺牲性能的前提下减轻计算负担。虽然将MoE集成到FL微调中具有巨大潜力，但仍面临三个关键挑战：i）由于缺乏可靠的指标来衡量每个专家对本地微调性能的影响，为客户端选择合适的专家仍然困难；ii）客户端之间异构的计算资源严重阻碍了基于MoE的LLM微调，因为不同输入样本间动态的专家激活可能使资源受限的设备不堪重负；iii）客户端特定的专家子集和路由偏好会破坏全局聚合，其中未对齐的专家更新和不一致的门控网络会引入破坏性干扰。为应对这些挑战，我们提出了HFedMoE，一种基于MoE的异构联邦学习微调框架，它为每个客户端定制专家子集以实现计算高效的LLM微调。具体而言，HFedMoE根据专家对微调性能的贡献来识别其重要性，然后从信息瓶颈的角度自适应地选择专家子集，以匹配每个客户端的计算预算。我们还设计了一种稀疏感知的模型聚合策略，以重要性加权的方式聚合主动微调的专家及门控参数。大量实验表明，HFedMoE在训练精度和收敛速度上均优于当前最先进的基准方法。