Reconstructing dynamic objects from monocular videos is a severely underconstrained and challenging problem, and recent work has approached it in various directions. However, owing to the ill-posed nature of this problem, there has been no solution that can provide consistent, high-quality novel views from camera positions that are significantly different from the training views. In this work, we introduce Neural Parametric Gaussians (NPGs) to take on this challenge by imposing a two-stage approach: first, we fit a low-rank neural deformation model, which then is used as regularization for non-rigid reconstruction in the second stage. The first stage learns the object's deformations such that it preserves consistency in novel views. The second stage obtains high reconstruction quality by optimizing 3D Gaussians that are driven by the coarse model. To this end, we introduce a local 3D Gaussian representation, where temporally shared Gaussians are anchored in and deformed by local oriented volumes. The resulting combined model can be rendered as radiance fields, resulting in high-quality photo-realistic reconstructions of the non-rigidly deforming objects. We demonstrate that NPGs achieve superior results compared to previous works, especially in challenging scenarios with few multi-view cues.

翻译：从单目视频中重建动态物体是一个严重欠约束且极具挑战性的问题，近期研究已从多个方向展开探索。然而，由于该问题的病态本质，目前尚未有方法能够从显著偏离训练视角的相机位置生成一致且高质量的新视角。本文提出神经参数化高斯体（Neural Parametric Gaussians, NPGs），通过两阶段方法应对这一挑战：首先，拟合一个低秩神经形变模型，该模型在第二阶段作为非刚性重建的正则化项。第一阶段学习物体的形变模式，以保持新视角下的一致性；第二阶段通过优化由粗模型驱动的三维高斯体实现高质量重建。为此，我们引入局部三维高斯体表征，将时间共享的高斯体锚定于局部有向体素中并由其驱动形变。最终组合模型可渲染为辐射场，实现对非刚性形变物体的高质量照片级重建。实验表明，NPGs在缺乏多视角线索的困难场景中，相比现有方法取得了显著更优的结果。