Realistic object interactions are crucial for creating immersive virtual experiences, yet synthesizing realistic 3D object dynamics in response to novel interactions remains a significant challenge. Unlike unconditional or text-conditioned dynamics generation, action-conditioned dynamics requires perceiving the physical material properties of objects and grounding the 3D motion prediction on these properties, such as object stiffness. However, estimating physical material properties is an open problem due to the lack of material ground-truth data, as measuring these properties for real objects is highly difficult. We present PhysDreamer, a physics-based approach that endows static 3D objects with interactive dynamics by leveraging the object dynamics priors learned by video generation models. By distilling these priors, PhysDreamer enables the synthesis of realistic object responses to novel interactions, such as external forces or agent manipulations. We demonstrate our approach on diverse examples of elastic objects and evaluate the realism of the synthesized interactions through a user study. PhysDreamer takes a step towards more engaging and realistic virtual experiences by enabling static 3D objects to dynamically respond to interactive stimuli in a physically plausible manner. See our project page at https://physdreamer.github.io/.

翻译：真实的物体交互对于创造沉浸式虚拟体验至关重要，但针对新交互行为合成逼真的3D物体动力学仍是一项重大挑战。与无条件或文本条件驱动的动力学生成不同，动作条件驱动的动力学需要感知物体的物理材料属性，并基于这些属性（例如物体刚度）进行3D运动预测。然而，由于缺乏材料真实标注数据（测量真实物体的这些属性极为困难），物理材料属性的估计仍是一个开放性问题。我们提出PhysDreamer——一种基于物理的方法，通过利用视频生成模型学习的物体动力学先验知识，赋予静态3D物体交互式动力学能力。通过蒸馏这些先验知识，PhysDreamer能够合成物体对新交互（如外力或智能体操控）的逼真响应。我们在多个弹性物体示例上展示了该方法，并通过用户研究评估了合成交互的真实性。PhysDreamer通过使静态3D物体以物理合理的方式动态响应交互刺激，向构建更具沉浸感和真实感的虚拟体验迈出了一步。详见项目页面：https://physdreamer.github.io/。