Adversarial robustness of the neural network is a significant concern when it is applied to security-critical domains. In this situation, adversarial distillation is a promising option which aims to distill the robustness of the teacher network to improve the robustness of a small student network. Previous works pretrain the teacher network to make it robust against the adversarial examples aimed at itself. However, the adversarial examples are dependent on the parameters of the target network. The fixed teacher network inevitably degrades its robustness against the unseen transferred adversarial examples which target the parameters of the student network in the adversarial distillation process. We propose PeerAiD to make a peer network learn the adversarial examples of the student network instead of adversarial examples aimed at itself. PeerAiD is an adversarial distillation that trains the peer network and the student network simultaneously in order to specialize the peer network for defending the student network. We observe that such peer networks surpass the robustness of the pretrained robust teacher model against adversarial examples aimed at the student network. With this peer network and adversarial distillation, PeerAiD achieves significantly higher robustness of the student network with AutoAttack (AA) accuracy by up to 1.66%p and improves the natural accuracy of the student network by up to 4.72%p with ResNet-18 on TinyImageNet dataset. Code is available at https://github.com/jaewonalive/PeerAiD.

翻译：神经网络的对抗鲁棒性在其应用于安全关键领域时是一个重要问题。在此背景下，对抗性蒸馏成为一种有前景的方法，它旨在蒸馏教师网络的鲁棒性，以提升小型学生网络的鲁棒性。先前的研究通过预训练教师网络，使其能够抵御针对自身生成的对抗样本。然而，对抗样本依赖于目标网络的参数。在对抗性蒸馏过程中，固定的教师网络不可避免地会降低其对针对学生网络参数的未见迁移对抗样本的鲁棒性。我们提出PeerAiD，使同伴网络学习学生网络的对抗样本，而非针对自身生成的对抗样本。PeerAiD是一种对抗性蒸馏方法，它同时训练同伴网络和学生网络，以使同伴网络专门用于防御学生网络。我们观察到，这类同伴网络在应对针对学生网络的对抗样本时，其鲁棒性超过了预训练的鲁棒教师模型。借助这种同伴网络和对抗性蒸馏，PeerAiD显著提升了学生网络的鲁棒性，在TinyImageNet数据集上使用ResNet-18时，AutoAttack（AA）准确率最高提升1.66个百分点，自然准确率最高提升4.72个百分点。代码见https://github.com/jaewonalive/PeerAiD。