Multi-modal models have shown a promising capability to effectively integrate information from various sources, yet meanwhile, they are found vulnerable to pervasive perturbations, such as uni-modal attacks and missing conditions. To counter these perturbations, robust multi-modal representations are highly expected, which are positioned well away from the discriminative multi-modal decision boundary. In this paper, different from conventional empirical studies, we focus on a commonly used joint multi-modal framework and theoretically discover that larger uni-modal representation margins and more reliable integration for modalities are essential components for achieving higher robustness. This discovery can further explain the limitation of multi-modal robustness and the phenomenon that multi-modal models are often vulnerable to attacks on the specific modality. Moreover, our analysis reveals how the widespread issue, that the model has different preferences for modalities, limits the multi-modal robustness by influencing the essential components and could lead to attacks on the specific modality highly effective. Inspired by our theoretical finding, we introduce a training procedure called Certifiable Robust Multi-modal Training (CRMT), which can alleviate this influence from modality preference and explicitly regulate essential components to significantly improve robustness in a certifiable manner. Our method demonstrates substantial improvements in performance and robustness compared with existing methods. Furthermore, our training procedure can be easily extended to enhance other robust training strategies, highlighting its credibility and flexibility.

翻译：多模态模型虽展现了有效整合多源信息的潜力，但同时也被发现易受单模态攻击与缺失条件等普遍扰动的影响。为应对这些扰动，需要构建远离判别性多模态决策边界的鲁棒多模态表征。本文区别于传统实证研究，聚焦常用联合多模态框架，从理论上发现：更大的单模态表征间隔与更可靠的模态融合机制是实现高鲁棒性的关键要素。这一发现可进一步解释多模态鲁棒性的局限性——即多模态模型常对特定模态攻击脆弱的现象。此外，我们的分析揭示了广泛存在的模型模态偏好问题如何通过影响关键要素限制多模态鲁棒性，并导致针对特定模态的攻击高度有效。受理论发现启发，我们提出名为可认证鲁棒多模态训练（CRMT）的训练流程，该流程能有效缓解模态偏好的影响，以可认证方式显式调控关键要素，显著提升模型鲁棒性。相比现有方法，本方法在性能与鲁棒性上均取得显著提升。更重要的是，CRMT训练流程可轻松扩展至增强其他鲁棒训练策略，彰显其可靠性与灵活性。