Adversarial training is a practical approach for improving the robustness of deep neural networks against adversarial attacks. Although bringing reliable robustness, the performance toward clean examples is negatively affected after adversarial training, which means a trade-off exists between accuracy and robustness. Recently, some studies have tried to use knowledge distillation methods in adversarial training, achieving competitive performance in improving the robustness but the accuracy for clean samples is still limited. In this paper, to mitigate the accuracy-robustness trade-off, we introduce the Multi-Teacher Adversarial Robustness Distillation (MTARD) to guide the model's adversarial training process by applying a strong clean teacher and a strong robust teacher to handle the clean examples and adversarial examples, respectively. During the optimization process, to ensure that different teachers show similar knowledge scales, we design the Entropy-Based Balance algorithm to adjust the teacher's temperature and keep the teachers' information entropy consistent. Besides, to ensure that the student has a relatively consistent learning speed from multiple teachers, we propose the Normalization Loss Balance algorithm to adjust the learning weights of different types of knowledge. A series of experiments conducted on public datasets demonstrate that MTARD outperforms the state-of-the-art adversarial training and distillation methods against various adversarial attacks.

翻译：对抗训练是提升深度神经网络对抗攻击鲁棒性的实用方法。尽管能带来可靠的鲁棒性，但对抗训练后模型在干净样本上的性能会下降，这意味着准确率与鲁棒性之间存在权衡。近期部分研究尝试在对抗训练中引入知识蒸馏方法，在提升鲁棒性方面取得了竞争性表现，但干净样本的准确率仍受限。本文为缓解准确率-鲁棒性权衡问题，提出多教师对抗鲁棒性蒸馏（MTARD）方法，通过分别使用强干净教师和强鲁棒教师处理干净样本与对抗样本，引导模型的对抗训练过程。在优化过程中，为保证不同教师具有相似的知识尺度，我们设计基于熵的平衡算法来调节教师温度，使教师的信息熵保持一致。此外，为确保学生从多个教师处获得相对一致的学习速度，我们提出归一化损失平衡算法来调整不同类型知识的学习权重。在公开数据集上的系列实验表明，MTARD在多种对抗攻击下均优于当前最先进的对抗训练与蒸馏方法。