We propose VRL3, a powerful data-driven framework with a simple design for solving challenging visual deep reinforcement learning (DRL) tasks. We analyze a number of major obstacles in taking a data-driven approach, and present a suite of design principles, novel findings, and critical insights about data-driven visual DRL. Our framework has three stages: in stage 1, we leverage non-RL datasets (e.g. ImageNet) to learn task-agnostic visual representations; in stage 2, we use offline RL data (e.g. a limited number of expert demonstrations) to convert the task-agnostic representations into more powerful task-specific representations; in stage 3, we fine-tune the agent with online RL. On a set of challenging hand manipulation tasks with sparse reward and realistic visual inputs, compared to the previous SOTA, VRL3 achieves an average of 780% better sample efficiency. And on the hardest task, VRL3 is 1220% more sample efficient (2440% when using a wider encoder) and solves the task with only 10% of the computation. These significant results clearly demonstrate the great potential of data-driven deep reinforcement learning.

翻译：摘要：我们提出VRL3，一个设计简洁且强大的数据驱动框架，用于解决具有挑战性的视觉深度强化学习(DRL)任务。我们分析了采用数据驱动方法时面临的多项主要障碍，并提出了关于数据驱动视觉DRL的一系列设计原则、新颖发现及关键见解。该框架包含三个阶段：在第一阶段，我们利用非强化学习数据集（如ImageNet）学习任务无关的视觉表征；在第二阶段，我们使用离线强化学习数据（如少量专家演示）将任务无关表征转化为更具针对性的任务特定表征；在第三阶段，我们通过在线强化学习对智能体进行微调。在具有稀疏奖励和真实视觉输入的一系列具有挑战性的手部操控任务中，与先前最先进方法相比，VRL3实现了平均780%的样本效率提升。在难度最高的任务中，VRL3的样本效率提升了1220%（使用更宽编码器时提升2440%），且仅需10%的计算资源即可完成任务。这些显著成果充分展示了数据驱动深度强化学习的巨大潜力。