Visual dubbing is the process of generating lip motions of an actor in a video to synchronise with given audio. Recent advances have made progress towards this goal but have not been able to produce an approach suitable for mass adoption. Existing methods are split into either person-generic or person-specific models. Person-specific models produce results almost indistinguishable from reality but rely on long training times using large single-person datasets. Person-generic works have allowed for the visual dubbing of any video to any audio without further training, but these fail to capture the person-specific nuances and often suffer from visual artefacts. Our method, based on data-efficient neural rendering priors, overcomes the limitations of existing approaches. Our pipeline consists of learning a deferred neural rendering prior network and actor-specific adaptation using neural textures. This method allows for $\textbf{high-quality visual dubbing with just a few seconds of data}$, that enables video dubbing for any actor - from A-list celebrities to background actors. We show that we achieve state-of-the-art in terms of $\textbf{visual quality}$ and $\textbf{recognisability}$ both quantitatively, and qualitatively through two user studies. Our prior learning and adaptation method $\textbf{generalises to limited data}$ better and is more $\textbf{scalable}$ than existing person-specific models. Our experiments on real-world, limited data scenarios find that our model is preferred over all others. The project page may be found at https://dubbingforeveryone.github.io/

翻译：视觉配音是指生成视频中演员的唇部动作，使其与给定音频同步。近年来虽取得一定进展，但尚未形成适合大规模推广的方案。现有方法分为通用型与特定演员型两类。特定演员模型可生成近乎真实的视觉效果，但需要依赖大规模单人数据集进行长时间训练；通用型方法虽能实现任意视频与任意音频的免训练配音，却无法捕捉演员的个体特征，且常出现视觉伪影。我们的方法基于数据高效的神经渲染先验，突破现有技术局限。该流程包含延迟神经渲染先验网络学习与基于神经纹理的演员适配模块，仅需数秒数据即可实现高质量视觉配音，使从一线明星到群众演员的任何视频角色均可配音。通过两项用户研究的定量与定性评估证明，该方法在视觉质量与可辨识度上均达到最优水平。相比现有特定演员模型，我们的先验学习与适配方法在有限数据场景下泛化性更优、可扩展性更强。针对真实有限数据场景的实验表明，本模型获得最高偏好度。项目页面详见 https://dubbingforeveryone.github.io/