Audio-visual speech separation has gained significant traction in recent years due to its potential applications in various fields such as speech recognition, diarization, scene analysis and assistive technologies. Designing a lightweight audio-visual speech separation network is important for low-latency applications, but existing methods often require higher computational costs and more parameters to achieve better separation performance. In this paper, we present an audio-visual speech separation model called Top-Down-Fusion Net (TDFNet), a state-of-the-art (SOTA) model for audio-visual speech separation, which builds upon the architecture of TDANet, an audio-only speech separation method. TDANet serves as the architectural foundation for the auditory and visual networks within TDFNet, offering an efficient model with fewer parameters. On the LRS2-2Mix dataset, TDFNet achieves a performance increase of up to 10\% across all performance metrics compared with the previous SOTA method CTCNet. Remarkably, these results are achieved using fewer parameters and only 28\% of the multiply-accumulate operations (MACs) of CTCNet. In essence, our method presents a highly effective and efficient solution to the challenges of speech separation within the audio-visual domain, making significant strides in harnessing visual information optimally.

翻译：近年来，音视频语音分离因其在语音识别、说话人日志、场景分析及辅助技术等多个领域的广泛应用而备受关注。设计轻量化的音视频语音分离网络对于低延迟应用至关重要，但现有方法往往需要更高的计算成本和更多参数才能实现更优的分离性能。本文提出一种名为"自上而下融合网络"（TDFNet）的音视频语音分离模型，该模型基于纯音频语音分离方法TDANet的架构，是目前音视频语音分离领域的先进（SOTA）模型。TDFNet中的听觉网络与视觉网络均以TDANet为架构基础，从而实现了参数更少的高效模型。在LRS2-2Mix数据集上，与先前先进方法CTCNet相比，TDFNet在所有性能指标上均实现了高达10%的提升。值得关注的是，这些成果是在使用更少参数、仅消耗CTCNet量级运算量（MACs）28%的条件下取得的。本质上，本方法为音视频领域的语音分离挑战提供了高效且有效的解决方案，在最优利用视觉信息方面取得了重要进展。