Multimodal foundation models (MFMs) represent a significant advancement in artificial intelligence, combining diverse data modalities to enhance learning and understanding across a wide range of applications. However, this integration also brings unique safety and security challenges. In this paper, we conceptualize cybersafety and cybersecurity in the context of multimodal learning and present a comprehensive Systematization of Knowledge (SoK) to unify these concepts in MFMs, identifying key threats to these models. We propose a taxonomy framework grounded in information theory, evaluating and categorizing threats through the concepts of channel capacity, signal, noise, and bandwidth. This approach provides a novel framework that unifies model safety and system security in MFMs, offering a more comprehensive and actionable understanding of the risks involved. We used this to explore existing defense mechanisms, and identified gaps in current research - particularly, a lack of protection for alignment between modalities and a need for more systematic defense methods. Our work contributes to a deeper understanding of the security and safety landscape in MFMs, providing researchers and practitioners with valuable insights for improving the robustness and reliability of these models.

翻译：多模态基础模型（MFMs）代表了人工智能领域的重大进展，它通过整合多种数据模态来增强广泛应用中的学习与理解能力。然而，这种整合也带来了独特的安全与安保挑战。本文在多模态学习的背景下概念化了网络安保与网络安全，并提出了一套全面的知识体系化（SoK）框架，以统一MFMs中的这些概念，并识别出对这些模型的关键威胁。我们提出了一个基于信息论的分类框架，通过信道容量、信号、噪声和带宽等概念来评估和分类威胁。该方法提供了一个新颖的框架，将MFMs中的模型安全性与系统安全性统一起来，从而对相关风险提供了更全面且可操作的见解。我们运用此框架探索了现有的防御机制，并指出了当前研究中的不足——尤其是缺乏对模态间对齐的保护，以及需要更系统化的防御方法。我们的工作有助于更深入地理解MFMs中的安全与安保格局，为研究人员和实践者提供了宝贵的见解，以提升这些模型的鲁棒性和可靠性。