Modeling sounds emitted from physical object interactions is critical for immersive perceptual experiences in real and virtual worlds. Traditional methods of impact sound synthesis use physics simulation to obtain a set of physics parameters that could represent and synthesize the sound. However, they require fine details of both the object geometries and impact locations, which are rarely available in the real world and can not be applied to synthesize impact sounds from common videos. On the other hand, existing video-driven deep learning-based approaches could only capture the weak correspondence between visual content and impact sounds since they lack of physics knowledge. In this work, we propose a physics-driven diffusion model that can synthesize high-fidelity impact sound for a silent video clip. In addition to the video content, we propose to use additional physics priors to guide the impact sound synthesis procedure. The physics priors include both physics parameters that are directly estimated from noisy real-world impact sound examples without sophisticated setup and learned residual parameters that interpret the sound environment via neural networks. We further implement a novel diffusion model with specific training and inference strategies to combine physics priors and visual information for impact sound synthesis. Experimental results show that our model outperforms several existing systems in generating realistic impact sounds. More importantly, the physics-based representations are fully interpretable and transparent, thus enabling us to perform sound editing flexibly.

翻译：物理对象相互作用产生的声音建模对于现实与虚拟世界中的沉浸式感知体验至关重要。传统的撞击声合成方法通过物理模拟获取一组能够表征并合成声音的物理参数，但这类方法需要精确获取物体几何细节与撞击位置信息，这在真实世界中难以实现，因此无法应用于从普通视频合成撞击声。另一方面，现有基于视频驱动的深度学习方法因缺乏物理知识，仅能捕捉视觉内容与撞击声之间的弱相关性。本文提出了一种物理驱动的扩散模型，能够为无声视频片段合成高保真撞击声。除视频内容外，我们引入额外的物理先验来引导撞击声合成过程。这些物理先验包括两类：一类是通过对含噪真实世界撞击声样本直接估计（无需复杂装置）得到的物理参数，另一类是通过神经网络学习、用于解释声学环境的残差参数。我们进一步设计了具有特定训练与推理策略的新型扩散模型，将物理先验与视觉信息相结合用于撞击声合成。实验结果表明，本模型在生成逼真撞击声方面优于现有多种系统。更重要的是，基于物理的表征具有完全可解释性与透明性，从而实现了灵活的声音编辑功能。