Progress in deep learning highlights the tremendous potential of utilizing diverse robotic datasets for attaining effective generalization and makes it enticing to consider leveraging broad datasets for attaining robust generalization in robotic learning as well. However, in practice, we often want to learn a new skill in a new environment that is unlikely to be contained in the prior data. Therefore we ask: how can we leverage existing diverse offline datasets in combination with small amounts of task-specific data to solve new tasks, while still enjoying the generalization benefits of training on large amounts of data? In this paper, we demonstrate that end-to-end offline RL can be an effective approach for doing this, without the need for any representation learning or vision-based pre-training. We present pre-training for robots (PTR), a framework based on offline RL that attempts to effectively learn new tasks by combining pre-training on existing robotic datasets with rapid fine-tuning on a new task, with as few as 10 demonstrations. PTR utilizes an existing offline RL method, conservative Q-learning (CQL), but extends it to include several crucial design decisions that enable PTR to actually work and outperform a variety of prior methods. To our knowledge, PTR is the first RL method that succeeds at learning new tasks in a new domain on a real WidowX robot with as few as 10 task demonstrations, by effectively leveraging an existing dataset of diverse multi-task robot data collected in a variety of toy kitchens. We also demonstrate that PTR can enable effective autonomous fine-tuning and improvement in a handful of trials, without needing any demonstrations. An accompanying overview video can be found in the supplementary material and at thi URL: https://sites.google.com/view/ptr-final/

翻译：深度学习领域的进展彰显了利用多样化机器人数据集实现有效泛化的巨大潜力，并促使我们考虑通过广泛数据集来增强机器人学习的鲁棒泛化能力。然而实践中，我们常需在全新环境中学习新技能，而这可能未包含在先验数据中。因此本文提出：如何结合现有多样化离线数据集与少量任务特定数据来学习新任务，同时保留大规模数据训练带来的泛化优势？本文证明端到端离线强化学习是解决该问题的有效方法，无需任何表征学习或基于视觉的预训练。我们提出机器人预训练框架（PTR），这是一种基于离线强化学习的方法，通过将现有机器人数据集上的预训练与新任务的快速微调相结合（仅需10次示范），实现高效学习新任务。PTR采用现有离线强化学习方法——保守Q学习（CQL），但通过引入若干关键设计改进，使其真正奏效并超越多种先验方法。据我们所知，PTR是首个能在真实WidowX机器人上通过仅10次任务示范学习新域新任务的强化学习方法，其关键在于有效利用现有玩具厨房场景中收集的多样化多任务机器人数据集。我们同时证明PTR能在无需任何示范的情况下，通过少量试错实现自主微调与性能提升。配套概述视频见补充材料及以下链接：https://sites.google.com/view/ptr-final/