Data-driven approaches have revolutionized 3D vision, enabling transformers to effectively reconstruct and generate static 3D objects. However, generating simulative 4D dynamics -- realistic temporal deformations of static objects under various physical conditions -- remains challenging and often ad hoc, despite its importance in building comprehensive 3D world models. Most existing methods assume a predefined physical model and use system identification to estimate parameters, restricting these methods to specific categories and small-scale datasets. We propose that these restrictions can be overcome by learning a data-driven kinematic state parameterization for object-centric physical systems. Specifically, we learn both a latent space representing all possible states of the object and a decoder that maps any sampled latent to a plausibly deformed shape of the object. We refer to this parameterization as Neural Object Kinematics (NeuROK), and learn a transformer-based encoder-decoder model on a curated large-scale 4D dataset. This formulation and the learned model significantly simplify the generation of simulative dynamics since we only need to consider the dynamics within a low-dimensional latent space from the Lagrangian mechanics' perspective in classical physics. We demonstrate the effectiveness and generality of this neural simulation framework across diverse dynamic object types, showing clear advantages over prior works. Project page: https://chen-geng.com/neurok

翻译：数据驱动方法已彻底改变三维视觉领域，使Transformer能够有效重建和生成静态三维物体。然而，生成可仿真的四维动力学——静态物体在不同物理条件下随时间推移的逼真形变——尽管对构建全面的三维世界模型至关重要，却仍具挑战性且常需临时处理。现有方法大多预设物理模型，并通过系统辨识估算参数，导致其局限于特定类别和小规模数据集。我们提出，通过学习物体中心物理系统的数据驱动运动学状态参数化，可突破这些限制。具体而言，我们同时学习了表征物体所有可能状态的潜在空间，以及将任意采样潜在变量映射至物体合理形变形状的解码器。我们将该参数化称为神经物体运动学（NeuROK），并在精心整理的大规模四维数据集上训练基于Transformer的编码器-解码器模型。该公式化及学习模型显著简化了可仿真动力学的生成，因为我们只需从经典物理学中拉格朗日力学的视角，在低维潜在空间内考虑动力学过程。我们展示了该神经仿真框架在多种动态物体类型中的有效性与通用性，相较于先前工作具有明显优势。项目页面：https://chen-geng.com/neurok