Video is a promising source of knowledge for embodied agents to learn models of the world's dynamics. Large deep networks have become increasingly effective at modeling complex video data in a self-supervised manner, as evaluated by metrics based on human perceptual similarity or pixel-wise comparison. However, it remains unclear whether current metrics are accurate indicators of performance on downstream tasks. We find empirically that for planning robotic manipulation, existing metrics can be unreliable at predicting execution success. To address this, we propose a benchmark for action-conditioned video prediction in the form of a control benchmark that evaluates a given model for simulated robotic manipulation through sampling-based planning. Our benchmark, Video Prediction for Visual Planning ($VP^2$), includes simulated environments with 11 task categories and 310 task instance definitions, a full planning implementation, and training datasets containing scripted interaction trajectories for each task category. A central design goal of our benchmark is to expose a simple interface -- a single forward prediction call -- so it is straightforward to evaluate almost any action-conditioned video prediction model. We then leverage our benchmark to study the effects of scaling model size, quantity of training data, and model ensembling by analyzing five highly-performant video prediction models, finding that while scale can improve perceptual quality when modeling visually diverse settings, other attributes such as uncertainty awareness can also aid planning performance.

翻译：视频是具身智能体学习世界动态知识的重要信息来源。大型深度网络在自监督方式下日益有效地建模复杂视频数据，其性能通过基于人类感知相似性或像素级对比的指标进行评估。然而，当前指标是否能准确反映下游任务性能仍不明确。我们通过实验发现，在规划机器人操作任务中，现有指标可能无法可靠预测执行成功率。针对这一问题，我们提出了一种面向动作条件视频预测的基准测试框架——以控制任务形式评估给定模型在模拟机器人操作中基于采样的规划性能。该基准（Video Prediction for Visual Planning, $VP^2$）包含11个任务类别及310个任务实例定义的模拟环境、完整规划实现方案，以及为每个任务类别录制的脚本化交互轨迹训练数据集。我们的核心设计目标在于暴露简洁接口（即单次前向预测调用），使几乎所有动作条件视频预测模型都能便捷地参与评估。通过分析五个高性能视频预测模型，我们利用该基准研究了模型规模、训练数据量及模型集成对性能的影响，发现虽然规模扩展能提升视觉多样场景下的感知质量，但不确定性感知等其他属性同样有助于提高规划性能。