The scarcity of large-scale robotic data has motivated the repurposing of foundation models from other modalities for policy learning. In this work, we introduce PhysGen (Learning Physics from Pretrained Video Generation Models), a scalable continuous and sequential world interaction framework that leverages autoregressive video generation to solve robotic manipulation tasks. By treating the pretrained video model as a proxy for a physics simulator, PhysGen models the dynamic interplay between the external environment and robot actions. We introduce a multimodal continuous representation that unifies video and action into shared physical tokens, bridging the gap between discrete video generation and continuous robotic control. This approach enables the seamless transfer of implicit physical knowledge-such as object permanence and dynamics-from video pretraining to downstream manipulation.To ensure efficient convergence, we incorporate causal masking, inverse kinematics, Lookahead Multi-Token Prediction (L-MTP), and key-value (KV) caching. Experimental results on the Libero and ManiSkill benchmarks demonstrate that PhysGen consistently outperforms robust baselines, surpassing OpenVLA and WorldVLA by margins of 13.8% and 8.8%, respectively. Notably, in real-world scenarios, PhysGen matches the performance of large-scale action-pretrained models like $π_0$ without requiring prior action-specific pretraining, demonstrating superior capability in physically complex tasks such as grasping transparent objects. These findings validate the potential of extracting physical intuition from pretrained video generators to facilitate generalizable robotic manipulation.

翻译：大规模机器人数据的稀缺促使研究者将其他模态的基础模型重新用于策略学习。本文提出PhysGen（从预训练视频生成模型学习物理规律），一种可扩展的连续序贯世界交互框架，通过自回归视频生成解决机器人操作任务。通过将预训练视频模型视为物理仿真器的代理，PhysGen对外部环境与机器人动作之间的动态交互进行建模。我们引入一种多模态连续表示，将视频和动作统一到共享的物理标记中，弥合了离散视频生成与连续机器人控制之间的鸿沟。该方法能够将隐式物理知识（如物体恒存性和动力学）从视频预训练无缝迁移至下游操作任务。为确保高效收敛，我们整合了因果掩码、逆运动学、前瞻多标记预测（L-MTP）和键值缓存（KV caching）。在Libero和ManiSkill基准上的实验结果表明，PhysGen始终优于稳健基线，分别超过OpenVLA和WorldVLA 13.8%和8.8%。值得注意的是，在真实场景中，PhysGen无需先验动作特定预训练即可达到大规模动作预训练模型（如$π_0$）的性能，在抓取透明物体等物理复杂任务中展现出卓越能力。这些发现验证了从预训练视频生成器提取物理直觉以促进通用机器人操作的潜力。