Dynamic 3D interaction has been attracting a lot of attention recently. However, creating such 4D content remains challenging. One solution is to animate 3D scenes with physics-based simulation, which requires manually assigning precise physical properties to the object or the simulated results would become unnatural. Another solution is to learn the deformation of 3D objects with the distillation of video generative models, which, however, tends to produce 3D videos with small and discontinuous motions due to the inappropriate extraction and application of physics priors. In this work, to combine the strengths and complementing shortcomings of the above two solutions, we propose to learn the physical properties of a material field with video diffusion priors, and then utilize a physics-based Material-Point-Method (MPM) simulator to generate 4D content with realistic motions. In particular, we propose motion distillation sampling to emphasize video motion information during distillation. In addition, to facilitate the optimization, we further propose a KAN-based material field with frame boosting. Experimental results demonstrate that our method enjoys more realistic motions than state-of-the-arts do.

翻译：动态三维交互近年来备受关注。然而，创建此类四维内容仍具挑战性。一种解决方案是通过基于物理的仿真为三维场景添加动画，但这需要手动为物体分配精确的物理属性，否则仿真结果将失真。另一种方案是通过视频生成模型的蒸馏学习三维物体的形变，但由于物理先验提取与应用不当，该方法往往只能生成运动幅度小且不连续的三维视频。为融合上述两种方案的优势并弥补其不足，本研究提出通过视频扩散先验学习材料场的物理属性，进而利用基于物理的质点法（MPM）仿真器生成具有逼真运动的四维内容。具体而言，我们提出运动蒸馏采样方法，在蒸馏过程中强化视频运动信息。此外，为优化训练过程，我们进一步提出结合帧增强技术的KAN基材料场。实验结果表明，本方法生成的运动效果相比现有技术更为真实。