Targeted transfer-based attacks involving adversarial examples pose a significant threat to large visual-language models (VLMs). However, the state-of-the-art (SOTA) transfer-based attacks incur high costs due to excessive iteration counts. Furthermore, the generated adversarial examples exhibit pronounced adversarial noise and demonstrate limited efficacy in evading defense methods such as DiffPure. To address these issues, inspired by score matching, we introduce AdvDiffVLM, which utilizes diffusion models to generate natural, unrestricted adversarial examples. Specifically, AdvDiffVLM employs Adaptive Ensemble Gradient Estimation to modify the score during the diffusion model's reverse generation process, ensuring the adversarial examples produced contain natural adversarial semantics and thus possess enhanced transferability. Simultaneously, to enhance the quality of adversarial examples further, we employ the GradCAM-guided Mask method to disperse adversarial semantics throughout the image, rather than concentrating them in a specific area. Experimental results demonstrate that our method achieves a speedup ranging from 10X to 30X compared to existing transfer-based attack methods, while maintaining superior quality of adversarial examples. Additionally, the generated adversarial examples possess strong transferability and exhibit increased robustness against adversarial defense methods. Notably, AdvDiffVLM can successfully attack commercial VLMs, including GPT-4V, in a black-box manner.

翻译：目标迁移攻击中的对抗样本对大型视觉-语言模型构成严重威胁。然而，当前最先进的迁移攻击方法因迭代次数过多而导致成本高昂。此外，生成的对抗样本噪声显著，且在绕过诸如DiffPure等防御方法方面效果有限。为解决这些问题，受分数匹配启发，我们提出AdvDiffVLM，利用扩散模型生成自然无约束的对抗样本。具体而言，AdvDiffVLM采用自适应集成梯度估计在扩散模型反向生成过程中修改分数，确保生成的对抗样本包含自然的对抗语义，从而增强其迁移性。同时，为进一步提升对抗样本质量，我们采用GradCAM引导的掩码方法将对抗语义分散至图像全局，而非集中于特定区域。实验结果表明，与现有迁移攻击方法相比，我们的方法实现了10倍至30倍的加速，同时保持对抗样本的优越质量。此外，生成的对抗样本具有强迁移性，且对对抗防御方法展现出增强的鲁棒性。值得注意的是，AdvDiffVLM能够以黑盒方式成功攻击包括GPT-4V在内的商业视觉-语言模型。