Reinforcement learning (RL) requires either manually specifying a reward function, which is often infeasible, or learning a reward model from a large amount of human feedback, which is often very expensive. We study a more sample-efficient alternative: using pretrained vision-language models (VLMs) as zero-shot reward models (RMs) to specify tasks via natural language. We propose a natural and general approach to using VLMs as reward models, which we call VLM-RMs. We use VLM-RMs based on CLIP to train a MuJoCo humanoid to learn complex tasks without a manually specified reward function, such as kneeling, doing the splits, and sitting in a lotus position. For each of these tasks, we only provide a single sentence text prompt describing the desired task with minimal prompt engineering. We provide videos of the trained agents at: https://sites.google.com/view/vlm-rm. We can improve performance by providing a second ``baseline'' prompt and projecting out parts of the CLIP embedding space irrelevant to distinguish between goal and baseline. Further, we find a strong scaling effect for VLM-RMs: larger VLMs trained with more compute and data are better reward models. The failure modes of VLM-RMs we encountered are all related to known capability limitations of current VLMs, such as limited spatial reasoning ability or visually unrealistic environments that are far off-distribution for the VLM. We find that VLM-RMs are remarkably robust as long as the VLM is large enough. This suggests that future VLMs will become more and more useful reward models for a wide range of RL applications.

翻译：强化学习（RL）需要手动指定奖励函数（这通常不可行），或者从大量人类反馈中学习奖励模型（这通常成本高昂）。我们研究了一种更具样本效率的替代方案：利用预训练的视觉-语言模型（VLM）作为零样本奖励模型（RM），通过自然语言指定任务。我们提出了一种自然且通用的方法，将VLM用作奖励模型，称之为VLM-RMs。基于CLIP的VLM-RMs用于训练MuJoCo人形机器人，无需手动指定奖励函数即可学习复杂任务，例如跪地、劈叉和莲花坐姿。对于每个任务，我们仅提供一个描述所需任务的单句文本提示，且提示工程极简。我们提供了训练后智能体的视频，网址为：https://sites.google.com/view/vlm-rm。通过提供第二个"基线"提示，并投影掉CLIP嵌入空间中与区分目标和基线无关的部分，我们可以进一步提升性能。此外，我们发现VLM-RMs存在显著的规模缩放效应：经过更多计算和数据训练的更大规模VLM是更好的奖励模型。我们遇到的VLM-RMs失败模式均与当前VLM的已知能力局限相关，例如有限的空间推理能力或视觉上不真实的、远离VLM分布的环境。我们发现，只要VLM足够大，VLM-RMs就具有显著的鲁棒性。这表明未来的VLM将成为广泛RL应用中越来越有用的奖励模型。