Prompt-driven Video Segmentation Foundation Models (VSFMs) such as SAM2 are increasingly deployed in applications like autonomous driving and digital pathology, raising concerns about backdoor threats. Surprisingly, we find that directly transferring classic backdoor attacks (e.g., BadNet) to VSFMs is almost ineffective, with ASR below 5\%. To understand this, we study encoder gradients and attention maps and observe that conventional training keeps gradients for clean and triggered samples largely aligned, while attention still focuses on the true object, preventing the encoder from learning a distinct trigger-related representation. To address this challenge, we propose BadVSFM, the first backdoor framework tailored to prompt-driven VSFMs. BadVSFM uses a two-stage strategy: (1) steer the image encoder so triggered frames map to a designated target embedding while clean frames remain aligned with a clean reference encoder; (2) train the mask decoder so that, across prompt types, triggered frame-prompt pairs produce a shared target mask, while clean outputs stay close to a reference decoder. Extensive experiments on two datasets and five VSFMs show that BadVSFM achieves strong, controllable backdoor effects under diverse triggers and prompts while preserving clean segmentation quality. Ablations over losses, stages, targets, trigger settings, and poisoning rates demonstrate robustness to reasonable hyperparameter changes and confirm the necessity of the two-stage design. Finally, gradient-conflict analysis and attention visualizations show that BadVSFM separates triggered and clean representations and shifts attention to trigger regions, while four representative defenses remain largely ineffective, revealing an underexplored vulnerability in current VSFMs.

翻译：提示驱动视频分割基础模型（如SAM2）在自动驾驶和数字病理学等领域的应用日益广泛，引发了对其后门威胁的担忧。令人惊讶的是，我们发现直接将经典后门攻击（如BadNet）迁移至此类模型几乎无效，攻击成功率低于5%。为探究原因，我们研究了编码器梯度和注意力图，发现传统训练方法下干净样本与触发样本的梯度保持高度对齐，且注意力仍聚焦于真实目标，导致编码器无法学习与触发器相关的独特表征。为应对这一挑战，我们提出了首个针对提示驱动视频分割基础模型的后门框架BadVSFM。该框架采用两阶段策略：（1）引导图像编码器，使触发帧映射至指定目标嵌入，同时保持干净帧与参考编码器的对齐；（2）训练掩码解码器，使所有提示类型下的触发帧-提示对生成统一的目标掩码，而干净输出则保持接近参考解码器。在两个数据集和五种视频分割基础模型上的大量实验表明，BadVSFM能在保持干净样本分割质量的同时，在不同触发器和提示条件下实现强效且可控的后门效果。通过对损失函数、训练阶段、攻击目标、触发器设置及投毒率的消融实验，验证了该框架对合理超参数变化的鲁棒性，并证实了两阶段设计的必要性。最后，梯度冲突分析和注意力可视化表明，BadVSFM能有效分离触发样本与干净样本的表征，并将注意力转移至触发区域，而四种典型防御方法均未能有效应对，这揭示了当前视频分割基础模型中尚未被充分探索的安全漏洞。