Despite its remarkable success in zero-shot image-text matching, CLIP remains highly vulnerable to adversarial perturbations on images. As adversarial fine-tuning is prohibitively costly, recent works explore various test-time defense strategies; however, these approaches still exhibit limited robustness. In this work, we revisit this problem and propose a simple yet effective strategy: Augmentation-based Test-time Adversarial Correction (ATAC). Our method operates directly in the embedding space of CLIP, calculating augmentation-induced drift vectors to infer a semantic recovery direction and correcting the embedding based on the angular consistency of these latent drifts. Across a wide range of benchmarks, ATAC consistently achieves remarkably high robustness, surpassing that of previous state-of-the-art methods by nearly 50\% on average, all while requiring minimal computational overhead. Furthermore, ATAC retains state-of-the-art robustness in unconventional and extreme settings and even achieves nontrivial robustness against adaptive attacks. Our results demonstrate that ATAC is an efficient method in a novel paradigm for test-time adversarial defenses in the embedding space of CLIP. Code is available at: https://github.com/kylin0421/ATAC

翻译：尽管CLIP在零样本图像-文本匹配中取得了显著成功，但其仍极易受到图像上对抗性扰动的影响。由于对抗性微调成本过高，近期研究探索了多种测试时防御策略，然而这些方法的鲁棒性仍然有限。本文重新审视该问题，并提出一种简单而有效的策略：基于增强的测试时对抗校正方法（ATAC）。该方法直接在CLIP的嵌入空间中运行，通过计算增强引发的漂移向量推断语义恢复方向，并基于这些潜在漂移的角一致性校正嵌入。在广泛基准测试中，ATAC持续实现极高的鲁棒性，平均超越先前最先进方法近50%，同时仅需极低计算开销。此外，ATAC在非常规及极端场景中仍保持最先进鲁棒性，甚至具备对抗自适应攻击的非平凡鲁棒性。我们的结果表明，ATAC是一种在CLIP嵌入空间中实现测试时对抗防御的新型范式的有效方法。代码开源地址：https://github.com/kylin0421/ATAC