Recent studies have increasingly acknowledged the advantages of incorporating visual data into speech enhancement (SE) systems. In this paper, we introduce a novel audio-visual SE approach, termed DCUC-Net (deep complex U-Net with conformer network). The proposed DCUC-Net leverages complex domain features and a stack of conformer blocks. The encoder and decoder of DCUC-Net are designed using a complex U-Net-based framework. The audio and visual signals are processed using a complex encoder and a ResNet-18 model, respectively. These processed signals are then fused using the conformer blocks and transformed into enhanced speech waveforms via a complex decoder. The conformer blocks consist of a combination of self-attention mechanisms and convolutional operations, enabling DCUC-Net to effectively capture both global and local audio-visual dependencies. Our experimental results demonstrate the effectiveness of DCUC-Net, as it outperforms the baseline model from the COG-MHEAR AVSE Challenge 2023 by a notable margin of 0.14 in terms of PESQ. Additionally, the proposed DCUC-Net performs comparably to a state-of-the-art model and outperforms all other compared models on the Taiwan Mandarin speech with video (TMSV) dataset.

翻译：近期研究日益认识到将视觉数据融入语音增强（SE）系统的优势。本文提出了一种新颖的视听语音增强方法，命名为DCUC-Net（深度复数U-Net与Conformer网络）。所提出的DCUC-Net利用复数域特征和一系列Conformer模块。DCUC-Net的编码器和解码器基于复数U-Net框架设计，其中音频信号和视觉信号分别通过复数编码器和ResNet-18模型处理。处理后的信号通过Conformer模块融合，再经由复数解码器转换为增强后的语音波形。Conformer模块结合了自注意力机制与卷积操作，使DCUC-Net能够有效捕捉全局与局部的视听依赖关系。实验结果表明，DCUC-Net在COG-MHEAR AVSE挑战赛2023基线模型上取得了显著性能提升，PESQ指标提高了0.14。此外，所提出的DCUC-Net在台湾普通话视频（TMSV）数据集上的表现与最先进模型相当，并优于所有其他比较模型。