The transferability of adversarial examples is a key issue in the security of deep neural networks. The possibility of an adversarial example crafted for a source model fooling another targeted model makes the threat of adversarial attacks more realistic. Measuring transferability is a crucial problem, but the Attack Success Rate alone does not provide a sound evaluation. This paper proposes a new methodology for evaluating transferability by putting distortion in a central position. This new tool shows that transferable attacks may perform far worse than a black box attack if the attacker randomly picks the source model. To address this issue, we propose a new selection mechanism, called FiT, which aims at choosing the best source model with only a few preliminary queries to the target. Our experimental results show that FiT is highly effective at selecting the best source model for multiple scenarios such as single-model attacks, ensemble-model attacks and multiple attacks (Code available at: https://github.com/t-maho/transferability_measure_fit).

翻译：对抗样本的可迁移性是深度神经网络安全中的关键问题。针对源模型生成的对抗样本能欺骗另一个目标模型，这使得对抗攻击的威胁更加现实。评估可迁移性是一个关键问题，但仅依靠攻击成功率无法提供可靠的评估。本文通过将失真置于核心位置，提出了一种新的可迁移性评估方法。这一新工具表明，若攻击者随机选择源模型，可迁移攻击的性能可能远低于黑盒攻击。为解决该问题，我们提出了一种名为FiT的新型选择机制，旨在仅通过少量对目标模型的初步查询即可选择最佳源模型。实验结果表明，FiT在多种场景（如单模型攻击、集成模型攻击和多攻击）中选择最佳源模型方面具有高效性（代码地址：https://github.com/t-maho/transferability_measure_fit）。