Audio-visual speech recognition (AVSR) gains increasing attention from researchers as an important part of human-computer interaction. However, the existing available Mandarin audio-visual datasets are limited and lack the depth information. To address this issue, this work establishes the MAVD, a new large-scale Mandarin multimodal corpus comprising 12,484 utterances spoken by 64 native Chinese speakers. To ensure the dataset covers diverse real-world scenarios, a pipeline for cleaning and filtering the raw text material has been developed to create a well-balanced reading material. In particular, the latest data acquisition device of Microsoft, Azure Kinect is used to capture depth information in addition to the traditional audio signals and RGB images during data acquisition. We also provide a baseline experiment, which could be used to evaluate the effectiveness of the dataset. The dataset and code will be released at https://github.com/SpringHuo/MAVD.

翻译：视听语音识别（AVSR）作为人机交互的重要组成部分，日益受到研究者的关注。然而，现有可用的普通话视听数据集规模有限且缺乏深度信息。为解决该问题，本研究建立了MAVD——一个全新的大规模普通话多模态语料库，包含64位汉语母语者录制的12,484条话语。为确保数据集覆盖多样化的真实场景，我们开发了一套用于清洗和过滤原始文本材料的流水线，以构建内容均衡的朗读材料。特别地，在数据采集过程中，我们采用微软最新的数据采集设备Azure Kinect，在传统音频信号和RGB图像之外同步捕获深度信息。我们还提供了基线实验，可用于评估数据集的有效性。数据集和代码将在https://github.com/SpringHuo/MAVD 发布。