The integration of visual cues has revitalized the performance of the target speech extraction task, elevating it to the forefront of the field. Nevertheless, this multi-modal learning paradigm often encounters the challenge of modality imbalance. In audio-visual target speech extraction tasks, the audio modality tends to dominate, potentially overshadowing the importance of visual guidance. To tackle this issue, we propose AVSepChain, drawing inspiration from the speech chain concept. Our approach partitions the audio-visual target speech extraction task into two stages: speech perception and speech production. In the speech perception stage, audio serves as the dominant modality, while visual information acts as the conditional modality. Conversely, in the speech production stage, the roles are reversed. This transformation of modality status aims to alleviate the problem of modality imbalance. Additionally, we introduce a contrastive semantic matching loss to ensure that the semantic information conveyed by the generated speech aligns with the semantic information conveyed by lip movements during the speech production stage. Through extensive experiments conducted on multiple benchmark datasets for audio-visual target speech extraction, we showcase the superior performance achieved by our proposed method.

翻译：视觉线索的整合重振了目标语音提取任务的性能，使其跃升至该领域的前沿。然而，这种多模态学习范式常面临模态不平衡的挑战。在音视频目标语音提取任务中，音频模态往往占据主导地位，可能削弱视觉引导的重要性。为解决这一问题，我们受言语链概念的启发，提出AVSepChain方法。该方法将音视频目标语音提取任务划分为两个阶段：言语感知与言语生成。在言语感知阶段，音频作为主导模态，而视觉信息作为条件模态；反之，在言语生成阶段，两者的角色互换。这种模态状态的转换旨在缓解模态不平衡问题。此外，我们引入对比语义匹配损失，确保在言语生成阶段生成的语音所传达的语义信息与唇部运动所传达的语义信息保持一致。通过在多个音视频目标语音提取基准数据集上进行的大量实验，我们展示了所提方法的优越性能。