Acoustic matching aims to re-synthesize an audio clip to sound as if it were recorded in a target acoustic environment. Existing methods assume access to paired training data, where the audio is observed in both source and target environments, but this limits the diversity of training data or requires the use of simulated data or heuristics to create paired samples. We propose a self-supervised approach to visual acoustic matching where training samples include only the target scene image and audio -- without acoustically mismatched source audio for reference. Our approach jointly learns to disentangle room acoustics and re-synthesize audio into the target environment, via a conditional GAN framework and a novel metric that quantifies the level of residual acoustic information in the de-biased audio. Training with either in-the-wild web data or simulated data, we demonstrate it outperforms the state-of-the-art on multiple challenging datasets and a wide variety of real-world audio and environments.

翻译：声学匹配旨在重新合成音频片段，使其听起来仿佛在目标声学环境中录制。现有方法依赖于配对训练数据，即音频在源环境和目标环境中均被观测到，但这限制了训练数据的多样性，或需要借助模拟数据及启发式方法生成配对样本。我们提出了一种自监督的视觉声学匹配方法，其训练样本仅包含目标场景图像及音频——无需参考声学失配的源音频。我们的方法通过条件生成对抗网络框架，以及一种量化去偏音频中残余声学信息水平的新颖度量，联合学习解耦房间声学特性并重新合成目标环境下的音频。通过使用野外网络数据或模拟数据进行训练，我们证明了该方法在多个具有挑战性的数据集以及广泛的实际音频与环境中，均优于现有最先进技术。