The speaker extraction technique seeks to single out the voice of a target speaker from the interfering voices in a speech mixture. Typically an auxiliary reference of the target speaker is used to form voluntary attention. Either a pre-recorded utterance or a synchronized lip movement in a video clip can serve as the auxiliary reference. The use of visual cue is not only feasible, but also effective due to its noise robustness, and becoming popular. However, it is difficult to guarantee that such parallel visual cue is always available in real-world applications where visual occlusion or intermittent communication can occur. In this paper, we study the audio-visual speaker extraction algorithms with intermittent visual cue. We propose a joint speaker extraction and visual embedding inpainting framework to explore the mutual benefits. To encourage the interaction between the two tasks, they are performed alternately with an interlacing structure and optimized jointly. We also propose two types of visual inpainting losses and study our proposed method with two types of popularly used visual embeddings. The experimental results show that we outperform the baseline in terms of signal quality, perceptual quality, and intelligibility.

翻译：说话人提取技术旨在从混合语音中分离出目标说话人的声音。通常，利用目标说话人的辅助参考信息来形成定向注意力，该参考信息可以是预先录制的语音片段，也可以是视频片段中同步的唇部运动。使用视觉线索不仅可行，而且因其对噪声的鲁棒性而有效，正变得日益流行。然而，在现实应用中，视觉遮挡或间歇性通信可能导致这种并行视觉线索难以确保始终可用。本文研究了在视觉线索间歇性出现情况下的视听说话人提取算法。我们提出了一种联合说话人提取与视觉嵌入修复框架，以探索两者的协同效益。为促进两个任务之间的交互，我们采用交错结构交替执行这两项任务并进行联合优化。此外，我们提出了两种类型的视觉修复损失函数，并采用两种广泛使用的视觉嵌入方式对所提方法进行了研究。实验结果表明，我们的方法在信号质量、感知质量和可懂度方面均优于基线方法。