We introduce 4D Motion Scaffolds (MoSca), a neural information processing system designed to reconstruct and synthesize novel views of dynamic scenes from monocular videos captured casually in the wild. To address such a challenging and ill-posed inverse problem, we leverage prior knowledge from foundational vision models, lift the video data to a novel Motion Scaffold (MoSca) representation, which compactly and smoothly encodes the underlying motions / deformations. The scene geometry and appearance are then disentangled from the deformation field, and are encoded by globally fusing the Gaussians anchored onto the MoSca and optimized via Gaussian Splatting. Additionally, camera poses can be seamlessly initialized and refined during the dynamic rendering process, without the need for other pose estimation tools. Experiments demonstrate state-of-the-art performance on dynamic rendering benchmarks.

翻译：我们提出了4D运动支架（MoSca），这是一种神经信息处理系统，旨在从野外随意拍摄的单目视频中重建和合成动态场景的新视角。为应对这一具有挑战性且不适定的逆问题，我们利用基础视觉模型的先验知识，将视频数据提升至新颖的运动支架（MoSca）表示，该表示能紧凑而平滑地编码底层运动/形变。场景几何与外观随后从形变场中解耦，并通过全局融合锚定在MoSca上的高斯表示进行编码，并借助高斯泼溅进行优化。此外，相机位姿可在动态渲染过程中无缝初始化与优化，无需依赖其他位姿估计工具。实验表明，本方法在动态渲染基准测试中达到了最先进的性能。