Advanced Audio-Visual Speech Recognition (AVSR) systems have been observed to be sensitive to missing video frames, performing even worse than single-modality models. While applying the dropout technique to the video modality enhances robustness to missing frames, it simultaneously results in a performance loss when dealing with complete data input. In this paper, we investigate this contrasting phenomenon from the perspective of modality bias and reveal that an excessive modality bias on the audio caused by dropout is the underlying reason. Moreover, we present the Modality Bias Hypothesis (MBH) to systematically describe the relationship between modality bias and robustness against missing modality in multimodal systems. Building on these findings, we propose a novel Multimodal Distribution Approximation with Knowledge Distillation (MDA-KD) framework to reduce over-reliance on the audio modality and to maintain performance and robustness simultaneously. Finally, to address an entirely missing modality, we adopt adapters to dynamically switch decision strategies. The effectiveness of our proposed approach is evaluated and validated through a series of comprehensive experiments using the MISP2021 and MISP2022 datasets. Our code is available at https://github.com/dalision/ModalBiasAVSR

翻译：先进的音视频语音识别（AVSR）系统被观察到对视频帧缺失敏感，其性能甚至低于单模态模型。尽管在视频模态上应用Dropout技术能增强对缺失帧的鲁棒性，但这同时会导致处理完整数据输入时的性能损失。本文从模态偏差视角研究这一对比现象，揭示出Dropout导致的过度音频模态偏差是潜在原因。此外，我们提出模态偏差假设（MBH），系统描述多模态系统中模态偏差与缺失模态鲁棒性之间的关系。基于这些发现，我们提出新颖的基于知识蒸馏的多模态分布逼近（MDA-KD）框架，以减少对音频模态的过度依赖，同时保持性能与鲁棒性。最后，针对完全缺失模态的场景，我们采用适配器动态切换决策策略。通过使用MISP2021和MISP2022数据集进行的一系列综合实验，验证了所提方法的有效性。我们的代码开源在https://github.com/dalision/ModalBiasAVSR