Multimodal fusion of multiple heterogeneous and interconnected signals is a fundamental challenge in almost all multimodal problems and applications. In order to perform multimodal fusion, we need to understand the types of interactions that modalities can exhibit: how each modality individually provides information useful for a task and how this information changes in the presence of other modalities. In this paper, we perform a comparative study of how human annotators can be leveraged to annotate two categorizations of multimodal interactions: (1) partial labels, where different randomly assigned annotators annotate the label given the first, second, and both modalities, and (2) counterfactual labels, where the same annotator is tasked to annotate the label given the first modality before giving them the second modality and asking them to explicitly reason about how their answer changes, before proposing an alternative taxonomy based on (3) information decomposition, where annotators annotate the degrees of redundancy: the extent to which modalities individually and together give the same predictions on the task, uniqueness: the extent to which one modality enables a task prediction that the other does not, and synergy: the extent to which only both modalities enable one to make a prediction about the task that one would not otherwise make using either modality individually. Through extensive experiments and annotations, we highlight several opportunities and limitations of each approach and propose a method to automatically convert annotations of partial and counterfactual labels to information decomposition, yielding an accurate and efficient method for quantifying interactions in multimodal datasets.

翻译：多模态融合涉及多个异构且相互关联的信号，这几乎是所有多模态问题和应用中的根本性挑战。为实现多模态融合，我们需要理解模态间可能呈现的交互类型：每个模态如何单独提供对任务有用的信息，以及这些信息在其他模态存在时如何改变。本文针对如何利用人类标注者对多模态交互的两种分类进行对比研究：（1）部分标签——由随机分配的不同标注者分别根据第一模态、第二模态及双模态组合进行标注；（2）反事实标签——由同一标注者先根据第一模态标注，再获取第二模态后明确推理其答案如何变化，最终提出基于（3）信息分解的替代分类体系。在该体系下，标注者需标注冗余度（模态单独及联合时对任务预测结果的一致程度）、独特性（某一模态能实现而另一模态无法实现的预测能力）及协同性（仅当双模态联合时才能作出任何单一模态无法实现的预测能力）。通过大量实验与标注工作，我们揭示了每种方法的多项机遇与局限，并提出将部分标签与反事实标签标注自动转换为信息分解的方法，从而为量化多模态数据集中的交互提供准确高效的解决方案。