This paper studies the problem of learning a control policy without the need for interactions with the environment; instead, learning purely from an existing dataset. Prior work has demonstrated that offline learning algorithms (e.g., behavioural cloning and offline reinforcement learning) are more likely to discover a satisfactory policy when trained using high-quality expert data. However, many real-world/practical datasets can contain significant proportions of examples generated using low-skilled agents. Therefore, we propose a behaviour discriminator (BD) concept, a novel and simple data filtering approach based on semi-supervised learning, which can accurately discern expert data from a mixed-quality dataset. Our BD approach was used to pre-process the mixed-skill-level datasets from the Real Robot Challenge (RRC) III, an open competition requiring participants to solve several dexterous robotic manipulation tasks using offline learning methods; the new BD method allowed a standard behavioural cloning algorithm to outperform other more sophisticated offline learning algorithms. Moreover, we demonstrate that the new BD pre-processing method can be applied to a number of D4RL benchmark problems, improving the performance of multiple state-of-the-art offline reinforcement learning algorithms.

翻译：本文研究了无需与环境交互即可学习控制策略的问题，而是仅从现有数据集中进行学习。已有研究表明，离线学习算法（如行为克隆和离线强化学习）在使用高质量专家数据训练时，更可能发现令人满意的策略。然而，许多实际数据集可能包含大量由低技能智能体生成的样本。因此，我们提出了行为判别器（BD）概念——一种基于半监督学习的新型简易数据筛选方法，能够从混合质量的数据集中准确识别专家数据。我们的BD方法被用于预处理第三届真实机器人挑战赛（RRC III）中的混合技能水平数据集，该公开竞赛要求参与者使用离线学习方法解决多项灵巧机器人操作任务；这一新的BD方法使标准行为克隆算法能够超越其他更复杂的离线学习算法。此外，我们证明了这种新的BD预处理方法可应用于多个D4RL基准问题，从而提升了多种最新离线强化学习算法的性能。