Defending machine-learning (ML) models against white-box adversarial attacks has proven to be extremely difficult. Instead, recent work has proposed stateful defenses in an attempt to defend against a more restricted black-box attacker. These defenses operate by tracking a history of incoming model queries, and rejecting those that are suspiciously similar. The current state-of-the-art stateful defense Blacklight was proposed at USENIX Security '22 and claims to prevent nearly 100% of attacks on both the CIFAR10 and ImageNet datasets. In this paper, we observe that an attacker can significantly reduce the accuracy of a Blacklight-protected classifier (e.g., from 82.2% to 6.4% on CIFAR10) by simply adjusting the parameters of an existing black-box attack. Motivated by this surprising observation, since existing attacks were evaluated by the Blacklight authors, we provide a systematization of stateful defenses to understand why existing stateful defense models fail. Finally, we propose a stronger evaluation strategy for stateful defenses comprised of adaptive score and hard-label based black-box attacks. We use these attacks to successfully reduce even reconfigured versions of Blacklight to as low as 0% robust accuracy.

翻译：保护机器学习模型免受白盒对抗攻击已被证明极为困难。为此，近期研究提出采用有状态防御机制，旨在应对更具限制性的黑盒攻击者。此类防御通过追踪传入模型的查询历史，并拒绝那些异常相似的查询来运作。当前最先进的有状态防御方法Blacklight于USENIX Security '22会议上提出，宣称能在CIFAR10和ImageNet数据集上阻止近100%的攻击。在本文中，我们发现攻击者仅需调整现有黑盒攻击的参数，即可显著降低经Blacklight保护的分类器的准确率（例如，在CIFAR10上从82.2%降至6.4%）。鉴于Blacklight作者已对现有攻击进行了评估，这一惊人发现促使我们对有状态防御进行系统化梳理，以理解现有有状态防御模型失效的原因。最终，我们提出一种由自适应评分和基于硬标签的黑盒攻击组成的有状态防御更强评估策略。通过运用这些攻击，我们成功将即使经过重新配置的Blacklight版本的鲁棒准确率降低至0%。