Data heterogeneity among Federated Learning (FL) users poses a significant challenge, resulting in reduced global model performance. The community has designed various techniques to tackle this issue, among which Knowledge Distillation (KD)-based techniques are common. While these techniques effectively improve performance under high heterogeneity, they inadvertently cause higher accuracy degradation under model poisoning attacks (known as attack amplification). This paper presents a case study to reveal this critical vulnerability in KD-based FL systems. We show why KD causes this issue through empirical evidence and use it as motivation to design a hybrid distillation technique. We introduce a novel algorithm, Hybrid Knowledge Distillation for Robust and Accurate FL (HYDRA-FL), which reduces the impact of attacks in attack scenarios by offloading some of the KD loss to a shallow layer via an auxiliary classifier. We model HYDRA-FL as a generic framework and adapt it to two KD-based FL algorithms, FedNTD and MOON. Using these two as case studies, we demonstrate that our technique outperforms baselines in attack settings while maintaining comparable performance in benign settings.

翻译：联邦学习（FL）中用户间的数据异构性构成了重大挑战，导致全局模型性能下降。研究界已设计了多种技术应对此问题，其中基于知识蒸馏（KD）的技术较为常见。尽管这些技术在高异构性环境下能有效提升性能，但它们无意中会在模型投毒攻击下造成更严重的精度下降（即攻击放大效应）。本文通过案例研究揭示了基于KD的FL系统中这一关键脆弱性。我们通过实证证据阐释了KD导致该问题的原因，并以此为动机设计了一种混合蒸馏技术。我们提出了一种新颖算法——面向鲁棒且精确联邦学习的混合知识蒸馏（HYDRA-FL），该算法通过辅助分类器将部分KD损失卸载至浅层网络，从而降低攻击场景下的攻击影响。我们将HYDRA-FL建模为通用框架，并将其适配至两种基于KD的FL算法——FedNTD与MOON。以这两种算法作为案例研究，我们证明该技术在攻击环境下优于基线方法，同时在良性环境下保持相当的性能表现。