Vision-Language Models (VLMs) are trained on image-text pairs collected under canonical visual conditions and achieve strong performance on multimodal tasks. However, their robustness to real-world weather conditions, and the stability of cross-modal semantic alignment under such structured perturbations, remain insufficiently studied. In this paper, we focus on rainy scenarios and introduce the first adversarial framework that exploits realistic weather to attack VLMs, using a two-stage, parameterized perturbation model based on semantic decoupling to analyze rain-induced shifts in decision-making. In Stage 1, we model the global effects of rainfall by applying a low-dimensional global modulation to condition the embedding space and gradually weaken the original semantic decision boundaries. In Stage 2, we introduce structured rain variations by explicitly modeling multi-scale raindrop appearance and rainfall-induced illumination changes, and optimize the resulting non-differentiable weather space to induce stable semantic shifts. Operating in a non-pixel parameter space, our framework generates perturbations that are both physically grounded and interpretable. Experiments across multiple tasks show that even physically plausible, highly constrained weather perturbations can induce substantial semantic misalignment in mainstream VLMs, posing potential safety and reliability risks in real-world deployment. Ablations further confirm that illumination modeling and multi-scale raindrop structures are key drivers of these semantic shifts.

翻译：视觉语言模型（VLMs）通常在规范视觉条件下收集的图像-文本对上进行训练，并在多模态任务中表现出强大性能。然而，其对于真实世界天气条件的鲁棒性，以及在此类结构化扰动下跨模态语义对齐的稳定性，仍未得到充分研究。本文聚焦于雨天场景，首次提出利用真实天气条件攻击VLMs的对抗性框架，采用基于语义解耦的两阶段参数化扰动模型来分析降雨引发的决策偏移。在第一阶段，我们通过对嵌入空间施加低维全局调制来建模降雨的全局效应，逐步弱化原始语义决策边界。在第二阶段，我们通过显式建模多尺度雨滴外观及降雨引发的光照变化来引入结构化雨效变化，并优化由此产生的不可微天气空间以诱导稳定的语义偏移。该框架在非像素参数空间中运行，生成的扰动既具有物理基础又可解释。跨多个任务的实验表明，即使物理上合理且高度受限的天气扰动也能在主流的VLMs中引发显著的语义错位，在实际部署中构成潜在的安全性与可靠性风险。消融实验进一步证实，光照建模与多尺度雨滴结构是驱动这些语义偏移的关键因素。