Vision-language models (VLMs) such as CLIP have demonstrated remarkable zero-shot generalization, yet remain highly vulnerable to adversarial examples (AEs). While test-time defenses are promising, existing methods fail to provide sufficient robustness against strong attacks and are often hampered by high inference latency and task-specific applicability. To address these limitations, we start by investigating the intrinsic properties of AEs, which reveals that AEs exhibit severe feature inconsistency under progressive frequency attenuation. We further attribute this to the model's inherent spectral bias. Leveraging this insight, we propose an efficient test-time defense named Contrastive Spectral Rectification (CSR). CSR optimizes a rectification perturbation to realign the input with the natural manifold under a spectral-guided contrastive objective, which is applied input-adaptively. Extensive experiments across 16 classification benchmarks demonstrate that CSR outperforms the SOTA by an average of 18.1% against strong APGD with modest inference overhead. Furthermore, CSR exhibits broad applicability across diverse visual tasks. Code is available at https://github.com/Summu77/CSR.

翻译：视觉语言模型（VLMs）如CLIP展现出卓越的零样本泛化能力，但仍极易受到对抗样本（AEs）的攻击。尽管测试时防御方法具有前景，但现有方法无法对强攻击提供足够的鲁棒性，且常受限于高推理延迟和任务特定适用性。为解决这些局限，我们首先探究了AEs的内在特性，发现AEs在渐进频率衰减下表现出严重的特征不一致性，并将其归因于模型固有的光谱偏差。基于此洞察，我们提出了一种高效的测试时防御方法——对比光谱矫正（CSR）。CSR通过优化矫正扰动，在光谱引导的对比目标下将输入重新对齐到自然流形，并以输入自适应方式应用。在16个分类基准上的大量实验表明，CSR在适度推理开销下，针对强APGD攻击平均超越现有最优方法（SOTA）18.1%。此外，CSR在多种视觉任务中展现出广泛适用性。代码已开源：https://github.com/Summu77/CSR。