Enhancing the robustness of deep learning models, particularly in the realm of vision transformers (ViTs), is crucial for their real-world deployment. In this work, we provide a finetuning approach to enhance the robustness of vision transformers inspired by the concept of nullspace from linear algebra. Our investigation centers on whether a vision transformer can exhibit resilience to input variations akin to the nullspace property in linear mappings, implying that perturbations sampled from this nullspace do not influence the model's output when added to the input. Firstly, we show that for many pretrained ViTs, a non-trivial nullspace exists due to the presence of the patch embedding layer. Secondly, as nullspace is a concept associated with linear algebra, we demonstrate that it is possible to synthesize approximate nullspace elements for the non-linear blocks of ViTs employing an optimisation strategy. Finally, we propose a fine-tuning strategy for ViTs wherein we augment the training data with synthesized approximate nullspace noise. After finetuning, we find that the model demonstrates robustness to adversarial and natural image perbutations alike.

翻译：增强深度学习模型的鲁棒性，特别是在视觉Transformer（ViTs）领域，对于其实际部署至关重要。本文受线性代数中零空间概念的启发，提出了一种增强视觉Transformer鲁棒性的微调方法。我们的核心研究问题是：视觉Transformer能否具备类似线性映射中零空间性质的对输入变化的抗扰能力，即从该零空间中采样的扰动添加到输入后不会影响模型输出。首先，我们证明了许多预训练ViT由于存在图像块嵌入层而具有非平凡零空间。其次，由于零空间是线性代数相关概念，我们证明可以通过优化策略为ViT的非线性模块合成近似零空间元素。最后，我们提出一种ViT微调策略，将合成的近似零空间噪声增广到训练数据中。实验表明，经过微调后，模型对对抗性扰动和自然图像扰动均表现出鲁棒性。