As an emerging and vital topic for studying deep neural networks' vulnerability (DNNs), backdoor learning has attracted increasing interest in recent years, and many seminal backdoor attack and defense algorithms are being developed successively or concurrently, in the status of a rapid arms race. However, mainly due to the diverse settings, and the difficulties of implementation and reproducibility of existing works, there is a lack of a unified and standardized benchmark of backdoor learning, causing unfair comparisons, and unreliable conclusions (e.g., misleading, biased or even false conclusions). Consequently, it is difficult to evaluate the current progress and design the future development roadmap of this literature. To alleviate this dilemma, we build a comprehensive benchmark of backdoor learning called BackdoorBench. Our benchmark makes three valuable contributions to the research community. 1) We provide an integrated implementation of state-of-the-art (SOTA) backdoor learning algorithms (currently including 16 attack and 27 defense algorithms), based on an extensible modular-based codebase. 2) We conduct comprehensive evaluations of 12 attacks against 16 defenses, with 5 poisoning ratios, based on 4 models and 4 datasets, thus 11,492 pairs of evaluations in total. 3) Based on above evaluations, we present abundant analysis from 8 perspectives via 18 useful analysis tools, and provide several inspiring insights about backdoor learning. We hope that our efforts could build a solid foundation of backdoor learning to facilitate researchers to investigate existing algorithms, develop more innovative algorithms, and explore the intrinsic mechanism of backdoor learning. Finally, we have created a user-friendly website at http://backdoorbench.com, which collects all important information of BackdoorBench, including codebase, docs, leaderboard, and model Zoo.

翻译：作为研究深度神经网络（DNN）脆弱性的新兴关键方向，后门学习近年来受到日益广泛的关注。随着研究领域呈现出快速军备竞赛的态势，大量开创性的后门攻击与防御算法正相继或并行地涌现。然而，由于现有研究普遍采用多样化实验设置且存在实现与复现困难，后门学习领域至今缺乏统一标准化的基准平台，导致算法比较缺乏公平性，研究结论也常出现误导性、偏倚甚至错误的情况，严重阻碍了领域发展进程的客观评估与未来研究路线图的规划。为打破这一困境，我们构建了名为BackdoorBench的后门学习综合基准平台。该基准对学术界具有三大核心贡献：1）基于可扩展的模块化代码架构，集成实现了当前最先进的后门学习算法（目前包含16种攻击算法与27种防御算法）；2）在4种模型与4种数据集上，针对5种投毒比例开展系统性评估，覆盖12种攻击方法对抗16种防御策略，共计完成11,492组对比实验；3）基于上述评估结果，通过18种分析工具从8个维度生成深度分析报告，并提出多项启发性的后门学习洞见。我们期望这项工作能为后门学习领域奠定坚实基础，助力研究者深入理解现有算法、开发创新性方法、探索后门学习的内在机理。我们已建立用户友好型网站http://backdoorbench.com，该平台整合了代码库、文档、技术排行榜及模型动物园等所有核心资源。