Multimodal learning is of continued interest in artificial intelligence-based applications, motivated by the potential information gain from combining different data modalities. However, modalities observed in the source environment may differ from the modalities observed in the target environment due to multiple factors, including cost, hardware failure, or the perceived \textit{informativeness} of a given modality. This change in missingness patterns between the source and target environment has not been carefully studied. Na{ï}ve estimation of the information gain associated with including an additional modality without accounting for missingness may result in improper estimates of that modality's value in the target environment. We formalize the problem of missingness, demonstrate its ubiquity, and show that the subsequent distribution shift induces bias when the missingness process is not explicitly accounted for. To address this issue, we introduce ICYM2I (In Case You Multimodal Missed It), a framework for the evaluation of predictive performance and information gain under missingness through inverse probability weighting-based correction. We demonstrate the importance of the proposed adjustment to estimate information gain under missingness on synthetic, semi-synthetic, and real-world datasets.

翻译：多模态学习在基于人工智能的应用中持续受到关注，其动机在于结合不同数据模态可能带来的信息增益。然而，由于成本、硬件故障或对特定模态感知的 \textit{信息性} 等多种因素，源环境中观测到的模态可能与目标环境中观测到的模态存在差异。这种源环境与目标环境之间缺失模式的变化尚未得到深入研究。若在未考虑缺失性的情况下，对包含额外模态所关联的信息增益进行朴素估计，可能导致对该模态在目标环境中价值的错误评估。我们形式化了缺失性问题，论证了其普遍存在性，并表明当缺失过程未被明确考虑时，随之产生的分布偏移会引入偏差。为解决此问题，我们提出了 ICYM2I（In Case You Multimodal Missed It）框架，该框架通过基于逆概率加权的校正，用于评估缺失性下的预测性能与信息增益。我们在合成、半合成及真实世界数据集上，证明了所提出的调整对于估计缺失性下信息增益的重要性。