We explore the task of embodied view synthesis from monocular videos of deformable scenes. Given a minute-long RGBD video of people interacting with their pets, we render the scene from novel camera trajectories derived from in-scene motion of actors: (1) egocentric cameras that simulate the point of view of a target actor and (2) 3rd-person cameras that follow the actor. Building such a system requires reconstructing the root-body and articulated motion of each actor in the scene, as well as a scene representation that supports free-viewpoint synthesis. Longer videos are more likely to capture the scene from diverse viewpoints (which helps reconstruction) but are also more likely to contain larger motions (which complicates reconstruction). To address these challenges, we present Total-Recon, the first method to photorealistically reconstruct deformable scenes from long monocular RGBD videos. Crucially, to scale to long videos, our method hierarchically decomposes the scene motion into the motion of each object, which itself is decomposed into global root-body motion and local articulations. To quantify such "in-the-wild" reconstruction and view synthesis, we collect ground-truth data from a specialized stereo RGBD capture rig for 11 challenging videos, significantly outperforming prior art. Code, videos, and data can be found at https://andrewsonga.github.io/totalrecon .

翻译：摘要：我们探索了基于单目视频的可变形场景的具身视角合成任务。给定一段人与宠物互动的单分钟RGBD视频，我们从场景中演员的运动轨迹生成新颖的摄像机视角来渲染场景：（1）模拟目标演员视角的自我中心摄像机，（2）跟随演员的第三人称摄像机。构建这样的系统需要重建场景中每个演员的根身体及关节运动，以及支持自由视角合成的场景表示。较长时间的视频更可能从多样的视角捕捉场景（这有助于重建），但也更可能包含较大的运动（这使重建复杂化）。为应对这些挑战，我们提出了Total-Recon，这是首个从长单目RGBD视频中逼真重建可变形场景的方法。关键之处在于，为了扩展至长视频，我们的方法将场景运动分层分解为每个物体的运动，而每个物体的运动又被分解为全局的根身体运动和局部关节运动。为了量化这种"野外"重建与视角合成效果，我们利用专门的双目RGBD采集装置为11个具有挑战性的视频收集了真实数据，显著超越了现有技术。代码、视频及数据可在https://andrewsonga.github.io/totalrecon获取。