As an emerging approach to explore the vulnerability of deep neural networks (DNNs), backdoor learning has attracted increasing interest in recent years, and many seminal backdoor attack and defense algorithms are being developed successively or concurrently, in the status of a rapid arms race. However, mainly due to the diverse settings, and the difficulties of implementation and reproducibility of existing works, there is a lack of a unified and standardized benchmark of backdoor learning, causing unfair comparisons or unreliable conclusions (e.g., misleading, biased or even false conclusions). Consequently, it is difficult to evaluate the current progress and design the future development roadmap of this literature. To alleviate this dilemma, we build a comprehensive benchmark of backdoor learning called BackdoorBench. Our benchmark makes three valuable contributions to the research community. 1) We provide an integrated implementation of state-of-the-art (SOTA) backdoor learning algorithms (currently including 20 attack and 32 defense algorithms), based on an extensible modular-based codebase. 2) We conduct comprehensive evaluations with 5 poisoning ratios, based on 4 models and 4 datasets, leading to 11,492 pairs of attack-against-defense evaluations in total. 3) Based on above evaluations, we present abundant analysis from 10 perspectives via 18 useful analysis tools, and provide several inspiring insights about backdoor learning. We hope that our efforts could build a solid foundation of backdoor learning to facilitate researchers to investigate existing algorithms, develop more innovative algorithms, and explore the intrinsic mechanism of backdoor learning. Finally, we have created a user-friendly website at http://backdoorbench.com, which collects all important information of BackdoorBench, including codebase, docs, leaderboard, and model Zoo.

翻译：作为一种探索深度神经网络（DNN）脆弱性的新兴方法，后门学习近年来受到越来越多的关注，许多开创性的后门攻击与防御算法在快速发展的攻防竞赛中相继或同时被提出。然而，主要由于现有工作的设置多样、实现与复现困难，目前缺乏一个统一且标准化的后门学习基准，导致不公平的比较或不可靠的结论（例如误导性、有偏见甚至错误的结论）。因此，难以评估该领域的当前进展并设计未来的发展路线图。为缓解这一困境，我们构建了一个名为BackdoorBench的综合后门学习基准。我们的基准为研究社区做出了三项重要贡献。1）我们基于一个可扩展的模块化代码库，提供了最先进（SOTA）后门学习算法（目前包含20种攻击算法和32种防御算法）的集成实现。2）我们在4种模型和4个数据集上，基于5种投毒比例进行了全面评估，总计完成了11,492组攻防对抗评估。3）基于上述评估，我们通过18个实用分析工具从10个视角进行了丰富分析，并提供了关于后门学习的若干启发性见解。我们希望这项工作能为后门学习奠定坚实基础，助力研究者深入考察现有算法、开发更具创新性的算法，并探索后门学习的内在机制。最后，我们创建了用户友好网站 http://backdoorbench.com，该网站汇集了BackdoorBench的所有重要信息，包括代码库、文档、排行榜和模型库。