During multimodal model training and testing, certain data modalities may be absent due to sensor limitations, cost constraints, privacy concerns, or data loss, negatively affecting performance. Multimodal learning techniques designed to handle missing modalities can mitigate this by ensuring model robustness even when some modalities are unavailable. This survey reviews recent progress in Multimodal Learning with Missing Modality (MLMM), focusing on deep learning methods. It provides the first comprehensive survey that covers the motivation and distinctions between MLMM and standard multimodal learning setups, followed by a detailed analysis of current methods, applications, and datasets, concluding with challenges and future directions.
翻译:在多模态模型的训练与测试过程中,由于传感器限制、成本约束、隐私顾虑或数据丢失等原因,某些数据模态可能出现缺失,从而对模型性能产生负面影响。专门为处理缺失模态而设计的多模态学习技术能够缓解这一问题,确保即使在部分模态不可用的情况下模型仍保持鲁棒性。本综述回顾了缺失模态多模态学习领域的最新进展,重点关注深度学习方法。本文首次提供了全面综述,涵盖了MLMM与标准多模态学习设置的动机与区别,随后详细分析了现有方法、应用场景与数据集,最后总结了当前面临的挑战与未来研究方向。