We propose DAVIS, a Diffusion model-based Audio-VIusal Separation framework that solves the audio-visual sound source separation task through a generative manner. While existing discriminative methods that perform mask regression have made remarkable progress in this field, they face limitations in capturing the complex data distribution required for high-quality separation of sounds from diverse categories. In contrast, DAVIS leverages a generative diffusion model and a Separation U-Net to synthesize separated magnitudes starting from Gaussian noises, conditioned on both the audio mixture and the visual footage. With its generative objective, DAVIS is better suited to achieving the goal of high-quality sound separation across diverse categories. We compare DAVIS to existing state-of-the-art discriminative audio-visual separation methods on the domain-specific MUSIC dataset and the open-domain AVE dataset, and results show that DAVIS outperforms other methods in separation quality, demonstrating the advantages of our framework for tackling the audio-visual source separation task.

翻译：本文提出DAVIS——一个基于扩散模型的音视频分离框架，通过生成式方法解决音视频声源分离任务。尽管现有基于掩码回归的判别式方法在该领域取得了显著进展，但它们在捕捉高质量分离多样化类别声音所需的复杂数据分布方面存在局限。相比之下，DAVIS利用生成式扩散模型与分离U-Net，以高斯噪声为起点，基于音频混合信号与视觉片段条件合成分离后的幅度谱。凭借其生成式目标，DAVIS更适用于实现高质量多类别声音分离。我们将DAVIS与现有最先进的判别式音视频分离方法在领域专用MUSIC数据集及开放域AVE数据集上进行对比，结果表明DAVIS在分离质量上优于其他方法，彰显了本框架处理音视频声源分离任务的优势。