Sabre is a defense to adversarial examples that was accepted at IEEE S&P 2024. We first reveal significant flaws in the evaluation that point to clear signs of gradient masking. We then show the cause of this gradient masking: a bug in the original evaluation code. By fixing a single line of code in the original repository, we reduce Sabre's robust accuracy to 0%. In response to this, the authors modify the defense and introduce a new defense component not described in the original paper. But this fix contains a second bug; modifying one more line of code reduces robust accuracy to below baseline levels. After we released the first version of our paper online, the authors introduced another change to the defense; by commenting out one line of code during attack we reduce the robust accuracy to 0% again.

翻译：Sabre是一种针对对抗样本的防御方法，曾被IEEE S&P 2024会议接收。我们首先揭示了其评估过程中存在的重大缺陷，这些缺陷明确指向梯度掩蔽现象。随后我们证明这种梯度掩蔽的根源在于原始评估代码中存在的一个错误。通过修复原始代码库中的单行代码，我们将Sabre的鲁棒准确率降至0%。作为回应，原作者修改了防御方案并引入了一个原论文中未描述的新防御组件。但该修复包含第二个错误；再修改一行代码即可将鲁棒准确率降至基线水平以下。在我们发布论文初版后，作者对防御方案进行了第三次修改；通过在攻击过程中注释掉一行代码，我们再次将鲁棒准确率降至0%。