Recent advances in internet-scale video data pretraining have led to the development of text-to-video generative models that can create high-quality videos across a broad range of visual concepts, synthesize realistic motions and render complex objects. Hence, these generative models have the potential to become general-purpose simulators of the physical world. However, it is unclear how far we are from this goal with the existing text-to-video generative models. To this end, we present VideoPhy, a benchmark designed to assess whether the generated videos follow physical commonsense for real-world activities (e.g. marbles will roll down when placed on a slanted surface). Specifically, we curate diverse prompts that involve interactions between various material types in the physical world (e.g., solid-solid, solid-fluid, fluid-fluid). We then generate videos conditioned on these captions from diverse state-of-the-art text-to-video generative models, including open models (e.g., CogVideoX) and closed models (e.g., Lumiere, Dream Machine). Our human evaluation reveals that the existing models severely lack the ability to generate videos adhering to the given text prompts, while also lack physical commonsense. Specifically, the best performing model, CogVideoX-5B, generates videos that adhere to the caption and physical laws for 39.6% of the instances. VideoPhy thus highlights that the video generative models are far from accurately simulating the physical world. Finally, we propose an auto-evaluator, VideoCon-Physics, to assess the performance reliably for the newly released models.

翻译：近年来，互联网规模视频数据预训练的进展推动了文本到视频生成模型的发展，这些模型能够针对广泛的视觉概念生成高质量视频，合成逼真的运动并渲染复杂物体。因此，这些生成模型有潜力成为物理世界的通用模拟器。然而，现有的文本到视频生成模型距离这一目标还有多远尚不清楚。为此，我们提出了VideoPhy，一个旨在评估生成视频是否遵循现实世界活动物理常识的基准（例如，大理石放在倾斜表面上会向下滚动）。具体而言，我们策划了多样化的提示，涉及物理世界中不同材料类型之间的相互作用（例如，固体-固体、固体-流体、流体-流体）。然后，我们基于这些描述，从多样化的最先进文本到视频生成模型（包括开源模型如CogVideoX和闭源模型如Lumiere、Dream Machine）生成视频。我们的人工评估表明，现有模型严重缺乏生成符合给定文本提示视频的能力，同时也缺乏物理常识。具体来说，表现最佳的模型CogVideoX-5B，仅为39.6%的实例生成了符合描述和物理定律的视频。因此，VideoPhy凸显了视频生成模型在准确模拟物理世界方面仍有很大差距。最后，我们提出了一个自动评估器VideoCon-Physics，以可靠地评估新发布模型的性能。