Audio and visual modalities are inherently connected in speech signals: lip movements and facial expressions are correlated with speech sounds. This motivates studies that incorporate the visual modality to enhance an acoustic speech signal or even restore missing audio information. Specifically, this paper focuses on the problem of audio-visual speech inpainting, which is the task of synthesizing the speech in a corrupted audio segment in a way that it is consistent with the corresponding visual content and the uncorrupted audio context. We present an audio-visual transformer-based deep learning model that leverages visual cues that provide information about the content of the corrupted audio. It outperforms the previous state-of-the-art audio-visual model and audio-only baselines. We also show how visual features extracted with AV-HuBERT, a large audio-visual transformer for speech recognition, are suitable for synthesizing speech.

翻译：音频与视觉模态在语音信号中具有内在关联性：唇部运动及面部表情与语音声学特征存在相关。这促使我们探索融合视觉模态来增强声学语音信号甚至修复缺失音频信息的研究。具体而言，本文聚焦于音视频联合语音修复问题，即通过合成受损音频片段中的语音内容，使其与对应视觉信息及未受损音频上下文保持一致性。我们提出了一种基于音视频Transformer的深度学习模型，该模型利用视觉线索获取受损音频的内容信息。该模型在性能上超越了此前最优的音视频模型及纯音频基线方法。研究还表明，通过大规模音视频语音识别模型AV-HuBERT提取的视觉特征，能够有效支持语音合成任务。