Despite recent advances in improving the sample-efficiency of reinforcement learning (RL) algorithms, designing an RL algorithm that can be practically deployed in real-world environments remains a challenge. In this paper, we present Coarse-to-fine Reinforcement Learning (CRL), a framework that trains RL agents to zoom-into a continuous action space in a coarse-to-fine manner, enabling the use of stable, sample-efficient value-based RL algorithms for fine-grained continuous control tasks. Our key idea is to train agents that output actions by iterating the procedure of (i) discretizing the continuous action space into multiple intervals and (ii) selecting the interval with the highest Q-value to further discretize at the next level. We then introduce a concrete, value-based algorithm within the CRL framework called Coarse-to-fine Q-Network (CQN). Our experiments demonstrate that CQN significantly outperforms RL and behavior cloning baselines on 20 sparsely-rewarded RLBench manipulation tasks with a modest number of environment interactions and expert demonstrations. We also show that CQN robustly learns to solve real-world manipulation tasks within a few minutes of online training.

翻译：尽管近年来在提升强化学习（RL）算法的样本效率方面取得了进展，设计一种能够实际部署于真实环境的RL算法仍然是一个挑战。本文提出粗到细强化学习（CRL）框架，该框架训练RL智能体以粗到细的方式逐步细化连续动作空间，从而能够将稳定、样本高效的价值型RL算法应用于细粒度连续控制任务。我们的核心思想是训练智能体通过迭代以下步骤输出动作：（i）将连续动作空间离散化为多个区间；（ii）选择具有最高Q值的区间，并在下一层级进一步离散化。随后，我们在CRL框架内提出了一种具体的价值型算法——粗到细Q网络（CQN）。实验表明，在20项稀疏奖励的RLBench操作任务上，CQN仅需适度的环境交互和专家演示，其性能显著优于RL与行为克隆基线方法。我们还证明，CQN能够在几分钟的在线训练内稳健地学习解决真实世界的操作任务。