Byzantine-robust federated learning aims at mitigating Byzantine failures during the federated training process, where malicious participants may upload arbitrary local updates to the central server to degrade the performance of the global model. In recent years, several robust aggregation schemes have been proposed to defend against malicious updates from Byzantine clients and improve the robustness of federated learning. These solutions were claimed to be Byzantine-robust, under certain assumptions. Other than that, new attack strategies are emerging, striving to circumvent the defense schemes. However, there is a lack of systematic comparison and empirical study thereof. In this paper, we conduct an experimental study of Byzantine-robust aggregation schemes under different attacks using two popular algorithms in federated learning, FedSGD and FedAvg . We first survey existing Byzantine attack strategies and Byzantine-robust aggregation schemes that aim to defend against Byzantine attacks. We also propose a new scheme, ClippedClustering , to enhance the robustness of a clustering-based scheme by automatically clipping the updates. Then we provide an experimental evaluation of eight aggregation schemes in the scenario of five different Byzantine attacks. Our results show that these aggregation schemes sustain relatively high accuracy in some cases but are ineffective in others. In particular, our proposed ClippedClustering successfully defends against most attacks under independent and IID local datasets. However, when the local datasets are Non-IID, the performance of all the aggregation schemes significantly decreases. With Non-IID data, some of these aggregation schemes fail even in the complete absence of Byzantine clients. We conclude that the robustness of all the aggregation schemes is limited, highlighting the need for new defense strategies, in particular for Non-IID datasets.

翻译：拜占庭鲁棒联邦学习旨在缓解联邦训练过程中的拜占庭故障，即恶意参与者可能向中央服务器上传任意局部更新，以降低全局模型的性能。近年来，已有几种鲁棒聚合方案被提出，用于抵御拜占庭客户端的恶意更新并提升联邦学习的鲁棒性。这些方案声称在特定假设下具有拜占庭鲁棒性。与此同时，新型攻击策略不断涌现，试图绕过防御机制。然而，目前缺乏对这些方案的系统比较和实证研究。本文针对联邦学习中的两种流行算法FedSGD和FedAvg，在不同攻击下对拜占庭鲁棒聚合方案进行了实验研究。我们首先综述了现有的拜占庭攻击策略及旨在防御拜占庭攻击的鲁棒聚合方案，并提出了一种新方案ClippedClustering，通过自动裁剪更新来增强基于聚类的方案的鲁棒性。随后，我们在五种不同拜占庭攻击场景下对八种聚合方案进行了实验评估。结果表明，这些聚合方案在部分情况下保持了较高精度，但在其他情况下效果不佳。特别地，我们提出的ClippedClustering在独立同分布局部数据集下成功抵御了大多数攻击。然而，当局部数据集呈非独立同分布时，所有聚合方案的性能均显著下降。在非独立同分布数据下，即使完全没有拜占庭客户端，部分聚合方案也会失效。我们得出结论：所有聚合方案的鲁棒性均有限，这凸显了针对非独立同分布数据集开发新型防御策略的必要性。