Vision-language pre-training (VLP) models exhibit remarkable capabilities in comprehending both images and text, yet they remain susceptible to multimodal adversarial examples (AEs).Strengthening attacks and uncovering vulnerabilities, especially common issues in VLP models (e.g., high transferable AEs), can advance reliable and practical VLP models. A recent work (i.e., Set-level guidance attack) indicates that augmenting image-text pairs to increase AE diversity along the optimization path enhances the transferability of adversarial examples significantly. However, this approach predominantly emphasizes diversity around the online adversarial examples (i.e., AEs in the optimization period), leading to the risk of overfitting the victim model and affecting the transferability.In this study, we posit that the diversity of adversarial examples towards the clean input and online AEs are both pivotal for enhancing transferability across VLP models. Consequently, we propose using diversification along the intersection region of adversarial trajectory to expand the diversity of AEs.To fully leverage the interaction between modalities, we introduce text-guided adversarial example selection during optimization. Furthermore, to further mitigate the potential overfitting, we direct the adversarial text deviating from the last intersection region along the optimization path, rather than adversarial images as in existing methods.Extensive experiments affirm the effectiveness of our method in improving transferability across various VLP models and downstream vision-and-language tasks.

翻译：视觉语言预训练（VLP）模型在理解图像和文本方面展现出卓越能力，但其仍易受多模态对抗样本（AEs）的攻击。强化攻击并揭示漏洞，尤其是VLP模型中的共性问题（例如高可迁移性对抗样本），能够推动可靠且实用的VLP模型发展。近期研究（即集合级引导攻击）表明，通过增强图像-文本对以增加沿优化路径的对抗样本多样性，可显著提升对抗样本的可迁移性。然而，该方法主要侧重于在线对抗样本（即优化期间的对抗样本）周围的多样性，可能导致对受害模型的过拟合风险，从而影响可迁移性。本研究提出，朝向干净输入和在线对抗样本的对抗样本多样性对于提升跨VLP模型的可迁移性均至关重要。因此，我们提出利用对抗轨迹交集区域的多样化来扩展对抗样本的多样性。为充分利用模态间的交互作用，我们在优化过程中引入了文本引导的对抗样本选择机制。此外，为进一步缓解潜在的过拟合问题，我们引导对抗文本沿优化路径偏离最后的交集区域，而非如现有方法那样处理对抗图像。大量实验证实了我们的方法在提升跨多种VLP模型及下游视觉-语言任务可迁移性方面的有效性。