Constructing AI models that respond to text instructions is challenging, especially for sequential decision-making tasks. This work introduces an instruction-tuned Video Pretraining (VPT) model for Minecraft called STEVE-1, demonstrating that the unCLIP approach, utilized in DALL-E 2, is also effective for creating instruction-following sequential decision-making agents. STEVE-1 is trained in two steps: adapting the pretrained VPT model to follow commands in MineCLIP's latent space, then training a prior to predict latent codes from text. This allows us to finetune VPT through self-supervised behavioral cloning and hindsight relabeling, bypassing the need for costly human text annotations. By leveraging pretrained models like VPT and MineCLIP and employing best practices from text-conditioned image generation, STEVE-1 costs just $60 to train and can follow a wide range of short-horizon open-ended text and visual instructions in Minecraft. STEVE-1 sets a new bar for open-ended instruction following in Minecraft with low-level controls (mouse and keyboard) and raw pixel inputs, far outperforming previous baselines. We provide experimental evidence highlighting key factors for downstream performance, including pretraining, classifier-free guidance, and data scaling. All resources, including our model weights, training scripts, and evaluation tools are made available for further research.

翻译：构建能够响应文本指令的AI模型极具挑战性，尤其是在序列决策任务中。本文引入了一种面向Minecraft的指令微调视频预训练（VPT）模型，命名为STEVE-1，论证了DALL-E 2所采用的unCLIP方法在创建遵循指令的序列决策智能体方面同样有效。STEVE-1的训练分两步进行：首先将预训练的VPT模型适配至MineCLIP潜在空间以遵循指令，随后训练一个先验模型以从文本预测潜在编码。这使得我们能够通过自监督行为克隆与事后重标记技术微调VPT，从而避免昂贵的人工文本标注需求。通过利用VPT和MineCLIP等预训练模型，并借鉴文本条件图像生成的最佳实践，STEVE-1的训练成本仅需60美元，即可在Minecraft中遵循多种短时域开放式文本与视觉指令。基于低层级控制（鼠标与键盘）和原始像素输入，STEVE-1在Minecraft开放式指令遵循任务中树立了新的标杆，性能显著超越此前基线方法。我们通过实验证据揭示了影响下游性能的关键因素，包括预训练、无分类器引导及数据规模。所有资源（含模型权重、训练脚本及评估工具）均已开源以供进一步研究。