Federated learning (FL) facilitates distributed training across clients, safeguarding the privacy of their data. The inherent distributed structure of FL introduces vulnerabilities, especially from adversarial (Byzantine) clients aiming to skew local updates to their advantage. Despite the plethora of research focusing on Byzantine-resilient FL, the academic community has yet to establish a comprehensive benchmark suite, pivotal for impartial assessment and comparison of different techniques. This paper investigates existing techniques in Byzantine-resilient FL and introduces an open-source benchmark suite for convenient and fair performance comparisons. Our investigation begins with a systematic study of Byzantine attack and defense strategies. Subsequently, we present \ours, a scalable, extensible, and easily configurable benchmark suite that supports researchers and developers in efficiently implementing and validating novel strategies against baseline algorithms in Byzantine-resilient FL. The design of \ours incorporates key characteristics derived from our systematic study, encompassing the attacker's capabilities and knowledge, defense strategy categories, and factors influencing robustness. Blades contains built-in implementations of representative attack and defense strategies and offers user-friendly interfaces for seamlessly integrating new ideas.

翻译：联邦学习（FL）支持跨客户端进行分布式训练，保护其数据隐私。FL固有的分布式结构引入了脆弱性，尤其是来自敌意（拜占庭）客户端试图操纵本地更新以谋取私利的攻击。尽管大量研究聚焦于拜占庭鲁棒性联邦学习，学术界仍未建立起一套全面的基准测试套件——这对于公正评估和比较不同技术至关重要。本文系统研究了拜占庭鲁棒性联邦学习中的现有技术，并引入了一套开源基准测试套件，以促进便捷且公平的性能对比。我们首先系统性地研究了拜占庭攻击与防御策略。随后，我们提出了Blades——一个可扩展、可伸缩且易于配置的基准测试套件，支持研究人员和开发者在拜占庭鲁棒性联邦学习中高效实现并验证新策略，与基线算法进行对比。Blades的设计融合了从系统研究中提炼的关键特征，包括攻击者的能力与知识、防御策略类别以及影响鲁棒性的因素。该套件内置了代表性攻击与防御策略的实现，并提供用户友好接口，以便无缝集成新想法。