Recent advances in generative adversarial networks (GANs) have demonstrated the capabilities of generating stunning photo-realistic portrait images. While some prior works have applied such image GANs to unconditional 2D portrait video generation and static 3D portrait synthesis, there are few works successfully extending GANs for generating 3D-aware portrait videos. In this work, we propose PV3D, the first generative framework that can synthesize multi-view consistent portrait videos. Specifically, our method extends the recent static 3D-aware image GAN to the video domain by generalizing the 3D implicit neural representation to model the spatio-temporal space. To introduce motion dynamics to the generation process, we develop a motion generator by stacking multiple motion layers to generate motion features via modulated convolution. To alleviate motion ambiguities caused by camera/human motions, we propose a simple yet effective camera condition strategy for PV3D, enabling both temporal and multi-view consistent video generation. Moreover, PV3D introduces two discriminators for regularizing the spatial and temporal domains to ensure the plausibility of the generated portrait videos. These elaborated designs enable PV3D to generate 3D-aware motion-plausible portrait videos with high-quality appearance and geometry, significantly outperforming prior works. As a result, PV3D is able to support many downstream applications such as animating static portraits and view-consistent video motion editing. Code and models are released at https://showlab.github.io/pv3d.

翻译：近期生成对抗网络（GANs）的进展已展现出生成逼真照片级肖像图像的能力。尽管部分先前工作已将此类图像GAN应用于无条件二维肖像视频生成与静态三维肖像合成，但成功将GAN扩展到生成三维感知肖像视频的研究仍鲜有报道。本文提出首个能合成多视角一致肖像视频的生成框架PV3D。具体而言，该方法通过将三维隐式神经表示泛化到时空域建模，将近期静态三维感知图像GAN扩展至视频领域。为在生成过程中引入运动动态，我们设计了一种运动生成器，通过堆叠多个运动层并采用调制卷积生成运动特征。为缓解相机/人体运动导致的运动模糊性，我们提出一种简单而有效的相机条件策略，使PV3D能同时实现时序一致与多视角一致的视频生成。此外，PV3D引入双判别器分别对空间域与时间域进行正则化，确保生成肖像视频的合理性。这些精妙设计使PV3D能够生成具有高质量外观与几何结构的三维感知运动合理肖像视频，性能显著超越先前工作。最终，PV3D可支持静态肖像动画化、视角一致视频运动编辑等多种下游应用。代码与模型已发布于https://showlab.github.io/pv3d。