Multimodal learning often outperforms its unimodal counterparts by exploiting unimodal contributions and cross-modal interactions. However, focusing only on integrating multimodal features into a unified comprehensive representation overlooks the unimodal characteristics. In real data, the contributions of modalities can vary from instance to instance, and they often reinforce or conflict with each other. In this study, we introduce a novel \text{MultiModal} loss paradigm for multimodal learning, which subgroups instances according to their unimodal contributions. \text{MultiModal} loss can prevent inefficient learning caused by overfitting and efficiently optimize multimodal models. On synthetic data, \text{MultiModal} loss demonstrates improved classification performance by subgrouping difficult instances within certain modalities. On four real multimodal datasets, our loss is empirically shown to improve the performance of recent models. Ablation studies verify the effectiveness of our loss. Additionally, we show that our loss generates a reliable prediction score for each modality, which is essential for subgrouping. Our \text{MultiModal} loss is a novel loss function to subgroup instances according to the contribution of modalities in multimodal learning and is applicable to a variety of multimodal models with unimodal decisions. Our code is available at https://github.com/SehwanMoon/MultiModalLoss.

翻译：多模态学习通过利用单模态贡献与跨模态交互，通常优于其单模态对应方法。然而，仅关注将多模态特征整合为统一的综合表示会忽略单模态特性。在实际数据中，各模态的贡献可能因实例而异，且常相互增强或冲突。本研究提出一种新颖的\text{MultiModal}损失范式用于多模态学习，该范式根据单模态贡献对实例进行子分组。\text{MultiModal}损失可防止因过拟合导致的低效学习，并高效优化多模态模型。在合成数据上，\text{MultiModal}损失通过子分组特定模态中的困难实例，展现了改进的分类性能。在四个真实多模态数据集上，实验证明该损失能提升近期模型的性能。消融研究验证了损失的有效性。此外，我们证明该损失能为每个模态生成可靠的预测分数，这对子分组至关重要。我们的\text{MultiModal}损失是一种新颖的损失函数，用于在多模态学习中根据模态贡献子分组实例，并适用于各类具有单模态决策的多模态模型。代码开源地址：https://github.com/SehwanMoon/MultiModalLoss。