Federated learning (FL) facilitates distributed training across different IoT and edge devices, safeguarding the privacy of their data. The inherent distributed structure of FL introduces vulnerabilities, especially from adversarial devices aiming to skew local updates to their advantage. Despite the plethora of research focusing on Byzantine-resilient FL, the academic community has yet to establish a comprehensive benchmark suite, pivotal for impartial assessment and comparison of different techniques. This paper presents Blades, a scalable, extensible, and easily configurable benchmark suite that supports researchers and developers in efficiently implementing and validating novel strategies against baseline algorithms in Byzantine-resilient FL. Blades contains built-in implementations of representative attack and defense strategies and offers a user-friendly interface that seamlessly integrates new ideas. Using Blades, we re-evaluate representative attacks and defenses on wide-ranging experimental configurations (approximately 1,500 trials in total). Through our extensive experiments, we gained new insights into FL robustness and highlighted previously overlooked limitations due to the absence of thorough evaluations and comparisons of baselines under various attack settings.

翻译：摘要：联邦学习（FL）支持跨不同物联网和边缘设备进行分布式训练，同时保护用户数据的隐私。FL固有的分布式结构引入了脆弱性，尤其是来自试图操纵局部更新以获取利益的对抗性设备。尽管已有大量研究关注拜占庭鲁棒的联邦学习，但学术界尚未建立一套全面的基准测试套件，而这对于公正评估和比较不同技术至关重要。本文提出Blades——一个可扩展、可配置且易于使用的基准测试套件，支持研究人员和开发者在拜占庭鲁棒联邦学习中高效实现并验证针对基线算法的新策略。Blades内置了代表性的攻击与防御策略实现，并提供用户友好的接口以无缝集成新方法。利用Blades，我们重新评估了多种代表性攻击与防御策略在广泛实验配置下的表现（总计约1500次试验）。通过大量实验，我们对FL的鲁棒性获得了新见解，并揭示了因缺乏全面评估及在不同攻击设置下基线比较而此前被忽视的局限性。