Adversarial examples are well-known tools to evaluate the vulnerability of deep neural networks (DNNs). Although lots of adversarial attack algorithms have been developed, it is still challenging in the practical scenario that the model's parameters and architectures are inaccessible to the attacker/evaluator, i.e., black-box adversarial attacks. Due to the practical importance, there has been rapid progress from recent algorithms, reflected by the quick increase in attack success rate and the quick decrease in query numbers to the target model. However, there is a lack of thorough evaluations and comparisons among these algorithms, causing difficulties of tracking the real progress, analyzing advantages and disadvantages of different technical routes, as well as designing future development roadmap of this field. Thus, in this work, we aim at building a comprehensive benchmark of black-box adversarial attacks, called BlackboxBench. It mainly provides: 1) a unified, extensible and modular-based codebase, implementing 25 query-based attack algorithms and 30 transfer-based attack algorithms; 2) comprehensive evaluations: we evaluate the implemented algorithms against several mainstreaming model architectures on 2 widely used datasets (CIFAR-10 and a subset of ImageNet), leading to 14,106 evaluations in total; 3) thorough analysis and new insights, as well analytical tools. The website and source codes of BlackboxBench are available at https://blackboxbench.github.io/ and https://github.com/SCLBD/BlackboxBench/, respectively.

翻译：对抗样本是评估深度神经网络脆弱性的重要工具。尽管已开发出大量对抗攻击算法，但在攻击者/评估者无法获取模型参数和架构的实际场景中（即黑盒对抗攻击）仍面临挑战。由于其实践重要性，近期算法取得了快速进展，表现为攻击成功率的显著提升和对目标模型查询次数的快速减少。然而，这些算法之间缺乏全面的评估与比较，导致难以追踪真实进展、分析不同技术路线的优缺点，以及设计该领域的未来发展路线图。为此，本研究旨在构建一个全面的黑盒对抗攻击基准测试——BlackboxBench。其主要提供：1）统一、可扩展且基于模块化的代码库，实现了25种基于查询的攻击算法和30种基于迁移的攻击算法；2）全面评估：我们在两个广泛使用的数据集（CIFAR-10和ImageNet子集）上对多种主流模型架构进行算法评估，总计完成14,106次评估；3）深入分析、新见解及分析工具。BlackboxBench的网站和源代码分别位于https://blackboxbench.github.io/ 和 https://github.com/SCLBD/BlackboxBench/。