As a privacy-preserving collaborative machine learning paradigm, federated learning (FL) has attracted significant interest from academia and the industry alike. To allow each data owner (a.k.a., FL clients) to train a heterogeneous and personalized local model based on its local data distribution, system resources and requirements on model structure, the field of model-heterogeneous personalized federated learning (MHPFL) has emerged. Existing MHPFL approaches either rely on the availability of a public dataset with special characteristics to facilitate knowledge transfer, incur high computation and communication costs, or face potential model leakage risks. To address these limitations, we propose a model-heterogeneous personalized Federated learning approach based on feature Extractor Sharing (pFedES). It incorporates a small homogeneous feature extractor into each client's heterogeneous local model. Clients train them via the proposed iterative learning method to enable the exchange of global generalized knowledge and local personalized knowledge. The small local homogeneous extractors produced after local training are uploaded to the FL server and for aggregation to facilitate easy knowledge sharing among clients. We theoretically prove that pFedES can converge over wall-to-wall time. Extensive experiments on two real-world datasets against six state-of-the-art methods demonstrate that pFedES builds the most accurate model, while incurring low communication and computation costs. Compared with the best-performing baseline, it achieves 1.61% higher test accuracy, while reducing communication and computation costs by 99.6% and 82.9%, respectively.

翻译：作为隐私保护的协作机器学习范式，联邦学习（FL）已引起学术界和工业界的广泛关注。为允许每个数据所有者（即FL客户端）基于其本地数据分布、系统资源和模型结构需求训练异构且个性化的本地模型，模型异构个性化联邦学习（MHPFL）领域应运而生。现有MHPFL方法要么依赖具有特殊特征的公开数据集以促进知识迁移，导致高计算与通信开销，要么面临模型泄露风险。为解决这些局限，我们提出一种基于特征提取器共享的模型异构个性化联邦学习方法（pFedES）。该方法在每个客户端的异构本地模型中嵌入一个同构的小型特征提取器。客户端通过提出的迭代学习方法联合训练这些模型，从而促进全局通用知识与本地个性化知识的交互。本地训练后产生的同构小型特征提取器被上传至FL服务器进行聚合，便于客户端间的知识共享。我们从理论上证明pFedES能在全程时间尺度上收敛。在两个真实数据集上针对六种最先进方法的广泛实验表明，pFedES在保持低通信与计算成本的同时构建了最精确的模型。与性能最优的基线方法相比，其测试准确率提升1.61%，同时通信成本降低99.6%，计算成本降低82.9%。