Adversarial attacks against Large Vision-Language Models (LVLMs) are crucial for exposing safety vulnerabilities in modern multimodal systems. Recent attacks based on input transformations, such as random cropping, suggest that spatially localized perturbations can be more effective than global image manipulation. However, randomly cropping the entire image is inherently stochastic and fails to use the limited per-pixel perturbation budget efficiently. We make two key observations: (i) regional attention scores are positively correlated with adversarial loss sensitivity, and (ii) attacking high-attention regions induces a structured redistribution of attention toward subsequent salient regions. Based on these findings, we propose Stage-wise Attention-Guided Attack (SAGA), an attention-guided framework that progressively concentrates perturbations on high-attention regions. SAGA enables more efficient use of constrained perturbation budgets, producing highly imperceptible adversarial examples while consistently achieving state-of-the-art attack success rates across ten LVLMs. The source code is available at https://github.com/jackwaky/SAGA.

翻译：针对大型视觉语言模型（LVLMs）的对抗攻击对于揭示现代多模态系统的安全漏洞至关重要。基于输入变换（如随机裁剪）的最新研究表明，空间局部化扰动可能比全局图像操作更有效。然而，对整个图像进行随机裁剪本质上具有随机性，且无法有效利用有限的每像素扰动预算。我们提出两个关键发现：（i）区域注意力分数与对抗损失敏感度呈正相关；（ii）攻击高注意力区域会引发注意力向后续显著区域的结构性重分布。基于这些发现，我们提出分阶段注意力引导攻击（SAGA），这是一种注意力引导框架，可逐步将扰动集中于高注意力区域。SAGA能更高效地利用受限的扰动预算，在十种LVLM上持续实现最先进的攻击成功率的同时，生成高度不易察觉的对抗样本。源代码发布于 https://github.com/jackwaky/SAGA。