Despite the remarkable success achieved by deep learning algorithms in various domains, such as computer vision, they remain vulnerable to adversarial perturbations. Adversarial Training (AT) stands out as one of the most effective solutions to address this issue; however, single-step AT can lead to Catastrophic Overfitting (CO). This scenario occurs when the adversarially trained network suddenly loses robustness against multi-step attacks like Projected Gradient Descent (PGD). Although several approaches have been proposed to address this problem in Convolutional Neural Networks (CNNs), we found out that they do not perform well when applied to Vision Transformers (ViTs). In this paper, we propose Blacksmith, a novel training strategy to overcome the CO problem, specifically in ViTs. Our approach utilizes either of PGD-2 or Fast Gradient Sign Method (FGSM) randomly in a mini-batch during the adversarial training of the neural network. This will increase the diversity of our training attacks, which could potentially mitigate the CO issue. To manage the increased training time resulting from this combination, we craft the PGD-2 attack based on only the first half of the layers, while FGSM is applied end-to-end. Through our experiments, we demonstrate that our novel method effectively prevents CO, achieves PGD-2 level performance, and outperforms other existing techniques including N-FGSM, which is the state-of-the-art method in fast training for CNNs.

翻译：尽管深度学习算法在计算机视觉等众多领域取得了显著成功，但其仍易受对抗性扰动的影响。对抗训练是解决此问题最有效的方法之一，但单步对抗训练可能导致灾难性过拟合。这种情况表现为，经过对抗训练的网络会突然丧失对投影梯度下降等多步攻击的鲁棒性。尽管针对卷积神经网络已提出多种解决方案，但我们发现这些方法在应用于视觉Transformer时效果不佳。本文提出一种名为Blacksmith的新型训练策略，专门用于克服视觉Transformer中的灾难性过拟合问题。本方法在神经网络对抗训练过程中，对每个小批量数据随机采用PGD-2或快速梯度符号法。此举可增加训练攻击的多样性，从而有效缓解灾难性过拟合。为控制该组合方法导致的训练时间增加，我们仅基于网络前半层构建PGD-2攻击，而FGSM则采用端到端方式执行。实验表明，我们的新方法能有效防止灾难性过拟合，达到PGD-2级性能，并优于包括N-FGSM（当前卷积神经网络快速训练的最先进方法）在内的现有技术。