View-predictive generative models provide strong priors for lifting object-centric images and videos into 3D and 4D through rendering and score distillation objectives. A question then remains: what about lifting complete multi-object dynamic scenes? There are two challenges in this direction: First, rendering error gradients are often insufficient to recover fast object motion, and second, view predictive generative models work much better for objects than whole scenes, so, score distillation objectives cannot currently be applied at the scene level directly. We present DreamScene4D, the first approach to generate 3D dynamic scenes of multiple objects from monocular videos via 360-degree novel view synthesis. Our key insight is a "decompose-recompose" approach that factorizes the video scene into the background and object tracks, while also factorizing object motion into 3 components: object-centric deformation, object-to-world-frame transformation, and camera motion. Such decomposition permits rendering error gradients and object view-predictive models to recover object 3D completions and deformations while bounding box tracks guide the large object movements in the scene. We show extensive results on challenging DAVIS, Kubric, and self-captured videos with quantitative comparisons and a user preference study. Besides 4D scene generation, DreamScene4D obtains accurate 2D persistent point track by projecting the inferred 3D trajectories to 2D. We will release our code and hope our work will stimulate more research on fine-grained 4D understanding from videos.

翻译：视角预测生成模型通过渲染与分数蒸馏目标，为将以物体为中心的图像与视频提升至三维与四维提供了强有力的先验。随之而来的一个问题是：如何提升完整的多物体动态场景？这一方向面临两大挑战：首先，渲染误差梯度通常不足以恢复快速的物体运动；其次，视角预测生成模型在物体层面效果显著优于整体场景层面，因此目前无法直接将分数蒸馏目标应用于场景级别。本文提出DreamScene4D，这是首个通过360度新视角合成从单目视频生成多物体三维动态场景的方法。我们的核心洞见在于一种“分解-重组”方法：将视频场景分解为背景与物体轨迹，同时将物体运动分解为三个分量——以物体为中心的形变、物体到世界坐标系的变换以及相机运动。这种分解使得渲染误差梯度与物体视角预测模型能够恢复物体的三维补全与形变，而边界框轨迹则指导场景中的大幅物体运动。我们在具有挑战性的DAVIS、Kubric及自采集视频上展示了大量结果，包括定量比较与用户偏好研究。除四维场景生成外，DreamScene4D通过将推断的三维轨迹投影至二维，可获得精确的二维持续点轨迹。我们将公开代码，并希望我们的工作能激发更多关于从视频中进行细粒度四维理解的研究。