Specifying reward signals that allow agents to learn complex behaviors is a long-standing challenge in reinforcement learning. A promising approach is to extract preferences for behaviors from unlabeled videos, which are widely available on the internet. We present Video Prediction Rewards (VIPER), an algorithm that leverages pretrained video prediction models as action-free reward signals for reinforcement learning. Specifically, we first train an autoregressive transformer on expert videos and then use the video prediction likelihoods as reward signals for a reinforcement learning agent. VIPER enables expert-level control without programmatic task rewards across a wide range of DMC, Atari, and RLBench tasks. Moreover, generalization of the video prediction model allows us to derive rewards for an out-of-distribution environment where no expert data is available, enabling cross-embodiment generalization for tabletop manipulation. We see our work as starting point for scalable reward specification from unlabeled videos that will benefit from the rapid advances in generative modeling. Source code and datasets are available on the project website: https://escontrela.me

翻译：在强化学习中，指定能使智能体学习复杂行为的奖励信号是一项长期挑战。一种有前景的方法是从互联网上广泛可得的无标签视频中提取行为偏好。我们提出视频预测奖励（VIPER）算法，该算法利用预训练的视频预测模型作为无动作奖励信号用于强化学习。具体而言，我们首先在专家视频上训练自回归Transformer模型，然后使用视频预测似然作为强化学习智能体的奖励信号。VIPER在DMC、Atari和RLBench的广泛任务中，无需程序化任务奖励即可实现专家级控制。此外，视频预测模型的泛化能力使我们能在无专家数据的分布外环境中推导奖励，从而实现桌面操作的跨具身泛化。我们视这项工作为从无标签视频中实现可扩展奖励规范的起点，这将得益于生成建模领域的快速进展。源代码和数据集可在项目网站上获取：https://escontrela.me