Reaching tasks with random targets and obstacles is a challenging task for robotic manipulators. In this study, we propose a novel model-free reinforcement learning approach based on proximal policy optimization (PPO) for training a deep policy to map the task space to the joint space of a 6-DoF manipulator. To facilitate the training process in a large workspace, we develop an efficient representation of environmental inputs and outputs. The calculation of the distance between obstacles and manipulator links is incorporated into the state representation using a geometry-based method. Additionally, to enhance the performance of the model in reaching tasks, we introduce the action ensembles method and design the policy to directly participate in value function updates in PPO. To overcome the challenges associated with training in real-robot environments, we develop a simulation environment in Gazebo to train the model as it produces a smaller Sim-to-Real gap compared to other simulators. However, training in Gazebo is time-intensive. To address this issue, we propose a Sim-to-Sim method to significantly reduce the training time. The trained model is then directly applied in a real-robot setup without fine-tuning. To evaluate the performance of the proposed approach, we perform several rounds of experiments in both simulated and real robots. We also compare the performance of the proposed approach with six baselines. The experimental results demonstrate the effectiveness of the proposed method in performing reaching tasks with and without obstacles. our method outperformed the selected baselines by a large margin in different reaching task scenarios. A video of these experiments has been attached to the paper as supplementary material.

翻译：针对随机目标与障碍物下的抓取任务对机器人操作臂构成的挑战，本研究提出一种新型无模型强化学习方法，基于近端策略优化（PPO）训练深度策略网络，将六自由度操作臂的任务空间映射至关节空间。为加速大工作空间内的训练进程，我们开发了高效的环境输入输出表征方法，通过几何算法将障碍物与操作臂连杆之间的距离计算融入状态表征。此外，为提升模型在抓取任务中的性能，我们引入动作集成方法，并设计策略直接参与PPO中的价值函数更新。为解决真实机器人环境的训练难题，我们在Gazebo中搭建仿真环境进行模型训练——该仿真器相比其他平台具有更小的仿真到现实迁移差距。然而Gazebo训练耗时较长，为此我们提出仿真到仿真方法显著缩短训练时间。训练后的模型无需微调即可直接部署于真实机器人系统。为评估方法性能，我们在仿真与真实机器人环境中开展多轮实验，并与六种基线方法进行对比。实验结果表明，本方法在有无障碍物的抓取任务中均表现优异，在不同任务场景下以显著优势超越所选基线方法。相关实验视频已作为补充材料附于论文中。