Despite considerable efforts on making them robust, real-world ML-based systems remain vulnerable to decision based attacks, as definitive proofs of their operational robustness have so far proven intractable. The canonical approach in robustness evaluation calls for adaptive attacks, that is with complete knowledge of the defense and tailored to bypass it. In this study, we introduce a more expansive notion of being adaptive and show how attacks but also defenses can benefit by it and by learning from each other through interaction. We propose and evaluate a framework for adaptively optimizing black-box attacks and defenses against each other through the competitive game they form. To reliably measure robustness, it is important to evaluate against realistic and worst-case attacks. We thus augment both attacks and the evasive arsenal at their disposal through adaptive control, and observe that the same can be done for defenses, before we evaluate them first apart and then jointly under a multi-agent perspective. We demonstrate that active defenses, which control how the system responds, are a necessary complement to model hardening when facing decision-based attacks; then how these defenses can be circumvented by adaptive attacks, only to finally elicit active and adaptive defenses. We validate our observations through a wide theoretical and empirical investigation to confirm that AI-enabled adversaries pose a considerable threat to black-box ML-based systems, rekindling the proverbial arms race where defenses have to be AI-enabled too. Succinctly, we address the challenges posed by adaptive adversaries and develop adaptive defenses, thereby laying out effective strategies in ensuring the robustness of ML-based systems deployed in the real-world.

翻译：尽管已付出大量努力以增强其鲁棒性，现实世界中基于机器学习的系统仍易受决策型攻击的影响，因为迄今为止，其运行鲁棒性的确定性证明仍难以实现。鲁棒性评估的经典方法要求采用自适应攻击，即攻击者完全了解防御机制并针对性地绕过它。在本研究中，我们引入了一种更为广泛的自适应概念，并展示了攻击与防御如何通过交互相互学习并从中获益。我们提出并评估了一个框架，用于通过攻击与防御之间形成的竞争博弈，自适应地优化针对彼此的黑盒攻击与防御。为了可靠地衡量鲁棒性，必须针对现实且最坏情况的攻击进行评估。因此，我们通过自适应控制增强了攻击及其可利用的规避手段，并观察到防御亦可采用相同方式增强；随后我们首先分别评估它们，再从多智能体视角进行联合评估。我们证明，在面对决策型攻击时，主动防御（即控制系统响应方式）是模型强化的必要补充；接着展示了这些防御如何被自适应攻击所规避，并最终催生出主动且自适应的防御。我们通过广泛的理论与实证研究验证了这些观察结果，证实具备AI能力的对手对基于机器学习的黑盒系统构成重大威胁，重新点燃了众所周知的军备竞赛，其中防御也必须具备AI能力。简言之，我们应对了自适应对手带来的挑战，并开发了自适应防御，从而为保障现实世界中部署的基于机器学习的系统的鲁棒性制定了有效策略。