Federated learning is an important privacy-preserving multi-party learning paradigm, involving collaborative learning with others and local updating on private data. Model heterogeneity and catastrophic forgetting are two crucial challenges, which greatly limit the applicability and generalizability. This paper presents a novel FCCL+, federated correlation and similarity learning with non-target distillation, facilitating the both intra-domain discriminability and inter-domain generalization. For heterogeneity issue, we leverage irrelevant unlabeled public data for communication between the heterogeneous participants. We construct cross-correlation matrix and align instance similarity distribution on both logits and feature levels, which effectively overcomes the communication barrier and improves the generalizable ability. For catastrophic forgetting in local updating stage, FCCL+ introduces Federated Non Target Distillation, which retains inter-domain knowledge while avoiding the optimization conflict issue, fulling distilling privileged inter-domain information through depicting posterior classes relation. Considering that there is no standard benchmark for evaluating existing heterogeneous federated learning under the same setting, we present a comprehensive benchmark with extensive representative methods under four domain shift scenarios, supporting both heterogeneous and homogeneous federated settings. Empirical results demonstrate the superiority of our method and the efficiency of modules on various scenarios.

翻译：联邦学习是一种重要的隐私保护多方学习范式，涉及与其他参与方的协作学习以及在私有数据上的本地更新。模型异构性和灾难性遗忘是两大关键挑战，极大地限制了其适用性和可推广性。本文提出了一种新颖的FCCL+方法，即带有非目标蒸馏的联邦相关性与相似性学习，以同时促进域内可区分性和域间泛化能力。针对异构性问题，我们利用无关的未标记公共数据实现异构参与方之间的通信。我们在logits和特征层面上构建跨相关矩阵并对齐实例相似性分布，这有效克服了通信障碍并提升了可推广能力。针对本地更新阶段的灾难性遗忘问题，FCCL+引入了联邦非目标蒸馏方法，该方法在保留域间知识的同时避免优化冲突问题，通过刻画后验类别关系充分提炼特权域间信息。考虑到现有异构联邦学习在统一设置下缺乏标准评估基准，我们构建了一个全面的基准，涵盖四种域偏移场景下的大量代表性方法，同时支持异构和同构联邦设置。实验结果证明了我们方法在各种场景下的优越性及各模块的有效性。