The imbalance problem is widespread in the field of machine learning, which also exists in multimodal learning areas caused by the intrinsic discrepancy between modalities of samples. Recent works have attempted to solve the modality imbalance problem from algorithm perspective, however, they do not fully analyze the influence of modality bias in datasets. Concretely, existing multimodal datasets are usually collected under specific tasks, where one modality tends to perform better than other ones in most conditions. In this work, to comprehensively explore the influence of modality bias, we first split existing datasets into different subsets by estimating sample-wise modality discrepancy. We surprisingly find that: the multimodal models with existing imbalance algorithms consistently perform worse than the unimodal one on specific subsets, in accordance with the modality bias. To further explore the influence of modality bias and analyze the effectiveness of existing imbalance algorithms, we build a balanced audiovisual dataset, with uniformly distributed modality discrepancy over the whole dataset. We then conduct extensive experiments to re-evaluate existing imbalance algorithms and draw some interesting findings: existing algorithms only provide a compromise between modalities and suffer from the large modality discrepancy of samples. We hope that these findings could facilitate future research on the modality imbalance problem.

翻译：不平衡问题在机器学习领域普遍存在，由于样本模态间的固有差异，该问题同样出现在多模态学习领域。近期研究试图从算法角度解决模态不平衡问题，但并未充分分析数据集中模态偏差的影响。具体而言，现有多模态数据集通常针对特定任务采集，在此类条件下某一模态往往比其他模态表现更优。为全面探究模态偏差的影响，我们首先通过估计样本级别的模态差异将现有数据集划分为不同子集。令人惊讶地发现：现有不平衡算法处理的多模态模型在特定子集上的表现始终逊于单模态模型，这与模态偏差的分布一致。为进一步探究模态偏差的影响并分析现有不平衡算法的有效性，我们构建了一个均衡视听数据集，该数据集在整个数据集范围内具有均匀分布的模态差异。随后我们开展大量实验重新评估现有不平衡算法，得出若干有趣结论：现有算法仅为模态间提供了折中方案，且难以应对样本较大的模态差异。我们期望这些发现能推动模态不平衡问题的后续研究。