We introduce a new method that generates photo-realistic humans under novel views and poses given a monocular video as input. Despite the significant progress recently on this topic, with several methods exploring shared canonical neural radiance fields in dynamic scene scenarios, learning a user-controlled model for unseen poses remains a challenging task. To tackle this problem, we introduce an effective method to a) integrate observations across several frames and b) encode the appearance at each individual frame. We accomplish this by utilizing both the human pose that models the body shape as well as point clouds that partially cover the human as input. Our approach simultaneously learns a shared set of latent codes anchored to the human pose among several frames, and an appearance-dependent code anchored to incomplete point clouds generated by each frame and its predicted depth. The former human pose-based code models the shape of the performer whereas the latter point cloud-based code predicts fine-level details and reasons about missing structures at the unseen poses. To further recover non-visible regions in query frames, we employ a temporal transformer to integrate features of points in query frames and tracked body points from automatically-selected key frames. Experiments on various sequences of dynamic humans from different datasets including ZJU-MoCap show that our method significantly outperforms existing approaches under unseen poses and novel views given monocular videos as input.

翻译：我们提出了一种新方法，能够以单目视频为输入，在未见过的视角和姿态下生成逼真的人体图像。尽管该领域近期取得了显著进展，已有若干方法在动态场景中探索共享规范神经辐射场，但学习用户可控模型以处理未见姿态仍是一项挑战。为解决此问题，我们引入了一种有效方法：a) 整合多帧观测信息，b) 编码每帧的外观。我们通过同时利用建模人体形状的姿态信息以及部分覆盖人体的点云作为输入来实现这一点。该方法联合学习多帧间锚定于人体姿态的共享隐编码集，以及锚定于每帧及其预测深度生成的不完整点云的外观依赖编码。前者基于人体姿态的编码建模表演者形状，后者基于点云的编码则预测精细细节并推断未见姿态下的缺失结构。为恢复查询帧中的不可见区域，我们采用时间变换器整合查询帧特征与从自动选取的关键帧中跟踪的身体点特征。在不同数据集（包括ZJU-MoCap）的多组动态人体序列上的实验表明，在单目视频输入下，我们的方法在未见姿态和新视角上显著优于现有方法。