Offline reinforcement learning has shown great promise in leveraging large pre-collected datasets for policy learning, allowing agents to forgo often-expensive online data collection. However, offline reinforcement learning from visual observations with continuous action spaces remains under-explored, with a limited understanding of the key challenges in this complex domain. In this paper, we establish simple baselines for continuous control in the visual domain and introduce a suite of benchmarking tasks for offline reinforcement learning from visual observations designed to better represent the data distributions present in real-world offline RL problems and guided by a set of desiderata for offline RL from visual observations, including robustness to visual distractions and visually identifiable changes in dynamics. Using this suite of benchmarking tasks, we show that simple modifications to two popular vision-based online reinforcement learning algorithms, DreamerV2 and DrQ-v2, suffice to outperform existing offline RL methods and establish competitive baselines for continuous control in the visual domain. We rigorously evaluate these algorithms and perform an empirical evaluation of the differences between state-of-the-art model-based and model-free offline RL methods for continuous control from visual observations. All code and data used in this evaluation are open-sourced to facilitate progress in this domain.

翻译：离线强化学习在利用大规模预收集数据集进行策略学习方面展现出巨大潜力，使智能体能够省去通常成本高昂的在线数据收集过程。然而，针对连续动作空间下视觉观测的离线强化学习仍处于探索不足的阶段，该复杂领域的关键挑战尚未得到充分理解。本文为视觉域中的连续控制任务建立了简单基线，并引入了一套专为视觉观测离线强化学习设计的基准测试任务集合——该集合更贴合真实世界离线强化学习问题中的数据分布特性，遵循视觉观测离线强化学习的一套期望准则（包括对视觉干扰的鲁棒性以及动态变化中可视觉识别的鲁棒性）。通过这套基准测试任务，我们发现对两种主流基于视觉的在线强化学习算法DreamerV2和DrQ-v2进行简单改进，即可超越现有离线强化学习方法，并为视觉域中的连续控制任务建立具有竞争力的基线。我们严格评估了这些算法，并对基于模型与无模型的最先进离线强化学习方法在视觉观测连续控制任务中的差异进行了实证分析。本研究使用的所有代码与数据均已开源，以促进该领域的进展。