Reinforcement learning is applied to solve actual complex tasks from high-dimensional, sensory inputs. The last decade has developed a long list of reinforcement learning algorithms. Recent progress benefits from deep learning for raw sensory signal representation. One question naturally arises: how well do they perform concerning different robotic manipulation tasks? Benchmarks use objective performance metrics to offer a scientific way to compare algorithms. In this paper, we present RMBench, the first benchmark for robotic manipulations, which have high-dimensional continuous action and state spaces. We implement and evaluate reinforcement learning algorithms that directly use observed pixels as inputs. We report their average performance and learning curves to show their performance and stability of training. Our study concludes that none of the studied algorithms can handle all tasks well, soft Actor-Critic outperforms most algorithms in average reward and stability, and an algorithm combined with data augmentation may facilitate learning policies. Our code is publicly available at https://github.com/xiangyanfei212/RMBench-2022, including all benchmark tasks and studied algorithms.

翻译：强化学习被应用于从高维感官输入解决实际复杂任务。过去十年间，研究者已开发出一系列强化学习算法。近期进展得益于深度学习对原始感官信号的表征能力。一个自然问题随之产生：这些算法在不同机器人操作任务中的表现如何？基准测试采用客观性能指标，为算法比较提供科学方法。本文提出首个面向机器人操作的基准测试平台RMBench，其任务具有高维连续动作空间与状态空间。我们实现并评估了直接以观测像素为输入的强化学习算法，通过汇报平均性能与学习曲线展示其训练效果与稳定性。研究表明：现有算法均无法完美应对所有任务；软演员-评论家算法在平均奖励与稳定性方面表现最优；结合数据增强的算法有助于策略学习。相关代码（含全部基准任务与算法）已开源至 https://github.com/xiangyanfei212/RMBench-2022。