Generating realistic audio for human interactions is important for many applications, such as creating sound effects for films or virtual reality games. Existing approaches implicitly assume total correspondence between the video and audio during training, yet many sounds happen off-screen and have weak to no correspondence with the visuals -- resulting in uncontrolled ambient sounds or hallucinations at test time. We propose a novel ambient-aware audio generation model, AV-LDM. We devise a novel audio-conditioning mechanism to learn to disentangle foreground action sounds from the ambient background sounds in in-the-wild training videos. Given a novel silent video, our model uses retrieval-augmented generation to create audio that matches the visual content both semantically and temporally. We train and evaluate our model on two in-the-wild egocentric video datasets Ego4D and EPIC-KITCHENS. Our model outperforms an array of existing methods, allows controllable generation of the ambient sound, and even shows promise for generalizing to computer graphics game clips. Overall, our work is the first to focus video-to-audio generation faithfully on the observed visual content despite training from uncurated clips with natural background sounds.

翻译：为人类交互生成逼真的音频对于许多应用至关重要，例如为电影或虚拟现实游戏创建音效。现有方法在训练时隐含地假设视频与音频之间存在完全对应关系，然而许多声音发生在屏幕外，与视觉内容的对应关系微弱甚至没有——这导致在测试时产生不受控制的环境声音或幻觉。我们提出了一种新颖的环境感知音频生成模型AV-LDM。我们设计了一种新颖的音频条件机制，旨在从真实世界训练视频中学习解耦前景动作声音与环境背景声音。给定一个新的无声视频，我们的模型使用检索增强生成技术来创建在语义和时间上都与视觉内容匹配的音频。我们在两个真实世界第一人称视角视频数据集Ego4D和EPIC-KITCHENS上训练和评估了我们的模型。我们的模型优于一系列现有方法，允许对环境声音进行可控生成，甚至显示出泛化到计算机图形游戏片段的潜力。总体而言，我们的工作是首个将视频到音频生成的焦点忠实于观察到的视觉内容的研究，尽管训练数据是未经剪辑的、带有自然背景声音的视频片段。