Large vision-language models (VLMs) such as GPT-4 have achieved unprecedented performance in response generation, especially with visual inputs, enabling more creative and adaptable interaction than large language models such as ChatGPT. Nonetheless, multimodal generation exacerbates safety concerns, since adversaries may successfully evade the entire system by subtly manipulating the most vulnerable modality (e.g., vision). To this end, we propose evaluating the robustness of open-source large VLMs in the most realistic and high-risk setting, where adversaries have only black-box system access and seek to deceive the model into returning the targeted responses. In particular, we first craft targeted adversarial examples against pretrained models such as CLIP and BLIP, and then transfer these adversarial examples to other VLMs such as MiniGPT-4, LLaVA, UniDiffuser, BLIP-2, and Img2Prompt. In addition, we observe that black-box queries on these VLMs can further improve the effectiveness of targeted evasion, resulting in a surprisingly high success rate for generating targeted responses. Our findings provide a quantitative understanding regarding the adversarial vulnerability of large VLMs and call for a more thorough examination of their potential security flaws before deployment in practice. Code is at https://github.com/yunqing-me/AttackVLM.

翻译：大型视觉-语言模型（VLMs，如GPT-4）在响应生成方面取得了前所未有的性能，尤其是结合视觉输入时，能够实现比大型语言模型（如ChatGPT）更具创造性和适应性的交互。然而，多模态生成加剧了安全隐患，因为攻击者可能通过巧妙操控最易受攻击的模态（例如视觉）来成功规避整个系统。为此，我们提出在最现实且高风险的情境下评估开源大型VLMs的鲁棒性，即攻击者仅具备黑盒系统访问权限，并试图欺骗模型返回目标响应。具体而言，我们首先针对预训练模型（如CLIP和BLIP）构造目标对抗样本，然后将这些对抗样本迁移至其他VLMs（如MiniGPT-4、LLaVA、UniDiffuser、BLIP-2和Img2Prompt）。此外，我们观察到对这些VLMs的黑盒查询能进一步提升目标规避的有效性，从而以惊人的高成功率生成目标响应。我们的发现定量揭示了大型VLMs的对抗脆弱性，并呼吁在实际部署前对其潜在安全缺陷进行更彻底的审查。代码地址：https://github.com/yunqing-me/AttackVLM。