Personalized federated learning (PFL) has been widely investigated to address the challenge of data heterogeneity, especially when a single generic model is inadequate in satisfying the diverse performance requirements of local clients simultaneously. Existing PFL methods are inherently based on the idea that the relations between the generic global and personalized local models are captured by the similarity of model weights. Such a similarity is primarily based on either partitioning the model architecture into generic versus personalized components, or modeling client relationships via model weights. To better capture similar (yet distinct) generic versus personalized model representations, we propose \textit{spectral distillation}, a novel distillation method based on model spectrum information. Building upon spectral distillation, we also introduce a co-distillation framework that establishes a two-way bridge between generic and personalized model training. Moreover, to utilize the local idle time in conventional PFL, we propose a wait-free local training protocol. Through extensive experiments on multiple datasets over diverse heterogeneous data settings, we demonstrate the outperformance and efficacy of our proposed spectral co-distillation method, as well as our wait-free training protocol.

翻译：个性化联邦学习（PFL）已被广泛研究以应对数据异构性挑战，尤其是当单一通用模型难以同时满足本地客户端多样化的性能需求时。现有PFL方法本质上基于"通用全局模型与个性化本地模型之间的关系可通过模型权重相似性捕捉"这一思路。此类相似性主要源于两种途径：将模型架构划分为通用组件与个性化组件，或通过模型权重建模客户端关系。为更精准地捕捉类似（却存在差异的）通用与个性化模型表示，我们提出基于模型频谱信息的**频谱蒸馏**——一种新型蒸馏方法。在频谱蒸馏基础上，我们进一步引入共蒸馏框架，在通用模型与个性化模型训练之间建立双向桥梁。此外，为利用传统PFL中的本地空闲时间，我们提出无等待本地训练协议。通过在多个数据集及多样异构数据设置下开展的大量实验，我们证明了所提出的频谱共蒸馏方法及无等待训练协议的性能优越性与有效性。