Transfer adversarial attacks raise critical security concerns in real-world, black-box scenarios. However, the actual progress of this field is difficult to assess due to two common limitations in existing evaluations. First, different methods are often not systematically and fairly evaluated in a one-to-one comparison. Second, only transferability is evaluated but another key attack property, stealthiness, is largely overlooked. In this work, we design good practices to address these limitations, and we present the first comprehensive evaluation of transfer attacks, covering 23 representative attacks against 9 defenses on ImageNet. In particular, we propose to categorize existing attacks into five categories, which enables our systematic category-wise analyses. These analyses lead to new findings that even challenge existing knowledge and also help determine the optimal attack hyperparameters for our attack-wise comprehensive evaluation. We also pay particular attention to stealthiness, by adopting diverse imperceptibility metrics and looking into new, finer-grained characteristics. Overall, our new insights into transferability and stealthiness lead to actionable good practices for future evaluations.

翻译：迁移对抗攻击在现实世界的黑盒场景中引发了严重的安全问题。然而，由于现有评估中普遍存在的两个局限，该领域的实际进展难以衡量。首先，不同方法往往未能在逐一比较中得到系统且公平的评估。其次，目前仅关注迁移性，而另一个关键攻击属性——隐蔽性——在很大程度上被忽视。在本工作中，我们设计了解决这些局限的良好实践，并首次对迁移攻击进行了全面评估，涵盖ImageNet上针对9种防御的23种代表性攻击。特别地，我们提出将现有攻击分为五类，从而实现系统的分类分析。这些分析产生了新发现，甚至挑战了现有知识，同时有助于确定攻击wise全面评估中的最优攻击超参数。我们还特别关注隐蔽性，采用了多种不可察觉性指标，并深入研究了新的、更细粒度的特征。总体而言，我们对迁移性和隐蔽性的新见解为未来评估提供了可操作的良好实践。