Audio-Visual Speech Recognition (AVSR) enhances speech recognition robustness by leveraging visual cues, while real-world scenarios remain challenging due to viewpoint variation, audio distortion, and visual occlusion, which degrade modality quality and increase audio-visual asynchrony. In this paper, we propose a novel Modality-aware Multi-view Self-supervised representation framework for robust Audio-Visual Speech Recognition (M2S-AVSR). First, we introduce a multi-view representation learning encoder to learn view-invariant visual speech representations. Next, we employ a modality-aware module that explicitly models modality quality and cross-modal synchrony to perform fine-grained modality-aware fusion, enabling fine-grained visual information injection during decoding. In addition, we release AISHELL8-RealScene, a public multi-scenario, multi-view conversational audio-visual dataset recorded in real-world environments, and establish a speech recognition benchmark on it. Experiments on English and Mandarin benchmarks demonstrate the effectiveness of the proposed method under challenging conditions. On LRS3, M2S-AVSR achieves up to 29.4% relative improvement under viewpoint perturbation and visual degradation settings. Our method also achieves new state-of-the-art performance on the MISP2021-AVSR test set. On AISHELL8-RealScene, it achieves the best result in outdoor scenes. The proposed method and dataset provide useful support for future research on robust speech and multimodal tasks under realistic conditions.

翻译：音视频语音识别（AVSR）通过利用视觉线索提升语音识别的鲁棒性，然而真实场景仍面临视角变化、音频失真和视觉遮挡等挑战，这些因素会降低模态质量并加剧音视频异步性。本文提出了一种新颖的模态感知多视角自监督表示框架，用于鲁棒音视频语音识别（M2S-AVSR）。首先，我们引入多视角表示学习编码器，以学习视角不变的视觉语音表示。其次，采用模态感知模块显式建模模态质量与跨模态同步性，实现细粒度的模态感知融合，从而在解码过程中注入精细的视觉信息。此外，我们发布了AISHELL8-RealScene——一个公开的多场景、多视角会话级音视频数据集（采集于真实环境），并在此基础上建立了语音识别基准。在英文和中文基准测试上的实验证明了该方法在挑战性条件下的有效性。在LRS3数据集上，M2S-AVSR在视角扰动与视觉退化设置下实现了最高29.4%的相对性能提升。我们的方法还在MISP2021-AVSR测试集上取得了新的最佳性能。在AISHELL8-RealScene上，该方法在户外场景中获得了最优结果。所提出的方法与数据集为未来真实条件下鲁棒语音及多模态任务研究提供了有力支持。