Reinforcement learning (RL) requires either manually specifying a reward function, which is often infeasible, or learning a reward model from a large amount of human feedback, which is often very expensive. We study a more sample-efficient alternative: using pretrained vision-language models (VLMs) as zero-shot reward models (RMs) to specify tasks via natural language. We propose a natural and general approach to using VLMs as reward models, which we call VLM-RMs. We use VLM-RMs based on CLIP to train a MuJoCo humanoid to learn complex tasks without a manually specified reward function, such as kneeling, doing the splits, and sitting in a lotus position. For each of these tasks, we only provide a single sentence text prompt describing the desired task with minimal prompt engineering. We provide videos of the trained agents at: https://sites.google.com/view/vlm-rm. We can improve performance by providing a second "baseline" prompt and projecting out parts of the CLIP embedding space irrelevant to distinguish between goal and baseline. Further, we find a strong scaling effect for VLM-RMs: larger VLMs trained with more compute and data are better reward models. The failure modes of VLM-RMs we encountered are all related to known capability limitations of current VLMs, such as limited spatial reasoning ability or visually unrealistic environments that are far off-distribution for the VLM. We find that VLM-RMs are remarkably robust as long as the VLM is large enough. This suggests that future VLMs will become more and more useful reward models for a wide range of RL applications.

翻译：强化学习（RL）通常需要手动指定奖励函数（这往往不可行），或从大量人类反馈中学习奖励模型（这通常代价高昂）。我们研究了一种更具样本效率的替代方案：利用预训练的视觉语言模型（VLM）作为零样本奖励模型（RM），通过自然语言指定任务。我们提出了一种自然且通用的基于VLM的奖励模型方法，称为VLM-RM。我们基于CLIP实现了VLM-RM，用于训练MuJoCo人形机器人执行无需人工指定奖励函数的复杂任务，例如跪姿、劈叉和莲花坐姿。对于每个任务，我们仅需提供一条描述目标任务的单句文本提示，且提示工程工作量极小。训练后智能体的演示视频可访问：https://sites.google.com/view/vlm-rm。通过提供第二个“基线”提示并投影去除CLIP嵌入空间中与区分目标和基线无关的部分，我们可进一步提升性能。此外，我们发现VLM-RM存在显著的规模效应：计算资源和数据量更大的VLM成为更优的奖励模型。我们遇到的VLM-RM失效模式均与当前VLM的已知能力限制相关，例如空间推理能力有限或视觉环境不真实（即与VLM训练数据分布严重偏离）。只要VLM规模足够大，VLM-RM便展现出惊人的鲁棒性。这表明未来VLM将成为广泛RL应用中越来越有价值的奖励模型。