Recent advances in generative adversarial networks (GANs) have demonstrated the capabilities of generating stunning photo-realistic portrait images. While some prior works have applied such image GANs to unconditional 2D portrait video generation and static 3D portrait synthesis, there are few works successfully extending GANs for generating 3D-aware portrait videos. In this work, we propose PV3D, the first generative framework that can synthesize multi-view consistent portrait videos. Specifically, our method extends the recent static 3D-aware image GAN to the video domain by generalizing the 3D implicit neural representation to model the spatio-temporal space. To introduce motion dynamics to the generation process, we develop a motion generator by stacking multiple motion layers to generate motion features via modulated convolution. To alleviate motion ambiguities caused by camera/human motions, we propose a simple yet effective camera condition strategy for PV3D, enabling both temporal and multi-view consistent video generation. Moreover, PV3D introduces two discriminators for regularizing the spatial and temporal domains to ensure the plausibility of the generated portrait videos. These elaborated designs enable PV3D to generate 3D-aware motion-plausible portrait videos with high-quality appearance and geometry, significantly outperforming prior works. As a result, PV3D is able to support many downstream applications such as animating static portraits and view-consistent video motion editing. Code and models are released at https://showlab.github.io/pv3d.

翻译：近期生成对抗网络（GANs）的进展已展现出生成逼真人像图像的能力。尽管部分先前工作已将该类图像生成对抗网络应用于无约束二维人像视频生成与静态三维人像合成，但鲜有研究成功拓展对抗网络以生成三维感知人像视频。本工作中，我们提出PV3D——首个能够合成多视角一致人像视频的生成框架。具体而言，本方法通过将三维隐式神经表示泛化至时空空间建模，将近期静态三维感知图像生成对抗网络拓展至视频领域。为在生成过程中引入运动动态，我们通过堆叠多个运动层构建运动生成器，利用调制卷积生成运动特征。为缓解相机/人体运动造成的运动模糊，我们提出一种简洁有效的相机条件策略，使PV3D能够同时生成时序一致与多视角一致的视频。此外，PV3D引入两个判别器分别对空间域与时间域进行正则化，确保生成人像视频的合理性。这些精心设计使PV3D能够生成具有高质量外观与几何结构的三维感知运动合理人像视频，显著优于先前工作。基于此，PV3D可支持诸多下游应用，如静态人像动画生成与视角一致视频运动编辑。代码与模型已发布于https://showlab.github.io/pv3d。