Specifying reward signals that allow agents to learn complex behaviors is a long-standing challenge in reinforcement learning. A promising approach is to extract preferences for behaviors from unlabeled videos, which are widely available on the internet. We present Video Prediction Rewards (VIPER), an algorithm that leverages pretrained video prediction models as action-free reward signals for reinforcement learning. Specifically, we first train an autoregressive transformer on expert videos and then use the video prediction likelihoods as reward signals for a reinforcement learning agent. VIPER enables expert-level control without programmatic task rewards across a wide range of DMC, Atari, and RLBench tasks. Moreover, generalization of the video prediction model allows us to derive rewards for an out-of-distribution environment where no expert data is available, enabling cross-embodiment generalization for tabletop manipulation. We see our work as starting point for scalable reward specification from unlabeled videos that will benefit from the rapid advances in generative modeling. Source code and datasets are available on the project website: https://escontrela.me/viper

翻译：在强化学习中，设定能够使智能体学习复杂行为的奖励信号是一个长期存在的挑战。一种有前景的方法是从互联网上广泛可用的无标签视频中提取行为偏好。我们提出了视频预测奖励（VIPER）算法，该算法利用预训练的视频预测模型作为强化学习的免动作奖励信号。具体而言，我们首先在专家视频上训练自回归Transformer，然后将视频预测似然作为强化学习智能体的奖励信号。VIPER能够在无需编程任务奖励的情况下，在多种DMC、Atari和RLBench任务中实现专家级控制。此外，视频预测模型的泛化能力使我们能够在无专家数据可用的分布外环境中推导出奖励信号，从而实现桌面操控的跨具身泛化。我们将这项工作视为从无标签视频中实现可扩展奖励规范的起点，这将受益于生成建模领域的快速进展。源代码和数据集可在项目网站上获取：https://escontrela.me/viper