Existing adversarial attacks for VLP models are mostly sample-specific, resulting in substantial computational overhead when scaled to large datasets or new scenarios. To overcome this limitation, we propose Hierarchical Refinement Attack (HRA), a multimodal universal attack framework for VLP models. For the image modality, we refine the optimization path by leveraging a temporal hierarchy of historical and estimated future gradients to avoid local minima and stabilize universal perturbation learning. For the text modality, it hierarchically models textual importance by considering both intra- and inter-sentence contributions to identify globally influential words, which are then used as universal text perturbations. Extensive experiments across various downstream tasks, VLP models, and datasets, demonstrate the superior transferability of the proposed universal multimodal attacks.

翻译：现有针对视觉-语言预训练模型的对抗攻击大多具有样本特异性，当扩展到大规模数据集或新场景时会产生巨大的计算开销。为克服这一局限，我们提出了分层细化攻击——一种面向视觉-语言预训练模型的多模态通用攻击框架。在图像模态方面，我们通过利用历史梯度与预估未来梯度构成的时间层次结构来优化路径，从而避免局部极小值并稳定通用扰动学习过程。在文本模态方面，该方法通过同时考虑句内贡献与句间贡献来分层建模文本重要性，以识别具有全局影响力的词汇，进而将其作为通用文本扰动。在多种下游任务、视觉-语言预训练模型及数据集上的大量实验表明，所提出的通用多模态攻击具有卓越的迁移性能。