Multimodal large language models (MLLMs) now appear in safety-critical applications, but the visual channel leaves them open to adversarial attacks that predominantly text-oriented safety alignment addresses only in part. Retraining a model for each new vulnerability class is usually too expensive to be practical. We report a comparative empirical evaluation of three inference-time defense methods and their combinations, run on eight models from the InternVL and Qwen-VL families across seven safety benchmarks that span four attack classes and total 9,000 evaluation samples. Every figure below comes from the same unified proxy classifier. Five findings emerge from the evaluation. First, within the evaluated models and benchmarks, no single defense dominates across all settings: what works depends on the model's baseline safety and on the attack type. Second, combining defenses directly drives benign-query over-refusal to 97-100% across all eight evaluated models, and SmoothVLM on its own reaches 99.2-100%. Third, a simple safety prompt keeps utility largely intact (0.0-18.2% over-refusal across all eight models, five of them below 7%, although two exceeded 15%) while still yielding moderate safety gains. Fourth, different attack classes expose different weaknesses across the evaluated setup, which is why multi-benchmark evaluation matters. Fifth, in a preliminary whitebox test on two models (n=20), text-level defenses suppressed a PGD visual attack that had succeeded without any defense: the defenses act at the output stage, where gradient optimization has limited direct leverage in the tested configuration. Read together, these results argue for adaptive defense selection rather than a single fixed defense configuration.

翻译：多模态大语言模型现已应用于安全关键场景，但其视觉通道容易遭受对抗攻击，而主要面向文本的安全性对齐仅能部分应对此类威胁。针对每种新型漏洞类别重新训练模型通常成本过高而难以实际应用。我们报告了对三种推理时防御方法及其组合的对比实证评估，在来自InternVL和Qwen-VL系列的八个模型上，跨越涵盖四类攻击的七个安全基准测试（总计9000个评估样本）进行实验。下文所有数据均来自统一的代理分类器。评估得出五项发现：第一，在所评估的模型与基准中，没有任何单一防御能在所有场景中占据主导地位——其效果取决于模型的基线安全水平及攻击类型；第二，直接组合防御导致所有八个评估模型的良性查询过度拒答率上升至97-100%，而SmoothVLM单算法即可达到99.2-100%；第三，简单安全提示在保持实用性的同时（八个模型中过度拒答率为0.0-18.2%，其中五个低于7%，两个超过15%），仍能带来中等程度的安全提升；第四，不同攻击类别在评估设置中暴露出不同弱点，这凸显了多基准评估的重要性；第五，在两个模型的初步白盒测试中（n=20），文本级防御成功抑制了原本无防御状态下成功的PGD视觉攻击：防御作用于输出阶段，在该测试配置下梯度优化的直接杠杆效应有限。综合上述结果，建议采用自适应防御选择策略，而非固定单一防御配置。