Reinforcement learning (RL) is a popular technique that allows an agent to learn by trial and error while interacting with a dynamic environment. The traditional Reinforcement Learning (RL) approach has been successful in learning and predicting Euclidean robotic manipulation skills such as positions, velocities, and forces. However, in robotics, it is common to encounter non-Euclidean data such as orientation or stiffness, and failing to account for their geometric nature can negatively impact learning accuracy and performance. In this paper, to address this challenge, we propose a novel framework for RL that leverages Riemannian geometry, which we call Geometric Reinforcement Learning (G-RL), to enable agents to learn robotic manipulation skills with non-Euclidean data. Specifically, G-RL utilizes the tangent space in two ways: a tangent space for parameterization and a local tangent space for mapping to a nonEuclidean manifold. The policy is learned in the parameterization tangent space, which remains constant throughout the training. The policy is then transferred to the local tangent space via parallel transport and projected onto the non-Euclidean manifold. The local tangent space changes over time to remain within the neighborhood of the current manifold point, reducing the approximation error. Therefore, by introducing a geometrically grounded pre- and post-processing step into the traditional RL pipeline, our G-RL framework enables several model-free algorithms designed for Euclidean space to learn from non-Euclidean data without modifications. Experimental results, obtained both in simulation and on a real robot, support our hypothesis that G-RL is more accurate and converges to a better solution than approximating non-Euclidean data.

翻译：强化学习是一种流行的技术，使智能体能够通过与动态环境交互进行试错学习。传统强化学习方法在学习和预测欧几里得空间中的机器人操控技能（如位置、速度和力）方面取得了成功。然而，在机器人学中，常会遇到非欧几里得数据（如方向或刚度），若忽视其几何特性，将对学习精度和性能产生负面影响。本文针对这一挑战，提出了一种基于黎曼几何的强化学习新框架——几何强化学习（G-RL），使智能体能够利用非欧几里得数据学习机器人操控技能。具体而言，G-RL以两种方式利用切空间：参数化切空间和局部切空间，用于映射到非欧几里得流形。策略在参数化切空间中学习，该空间在训练过程中保持不变。随后通过平行传输将策略迁移到局部切空间，并投影至非欧几里得流形。局部切空间随时间动态调整，始终位于当前流形点的邻域内，从而降低近似误差。因此，通过在传统强化学习流程中引入几何先验的前处理和后处理步骤，G-RL框架使多种面向欧几里得空间的无模型算法无需修改即可从非欧几里得数据中学习。仿真实验结果和真实机器人实验均支持我们的假设：与直接近似非欧几里得数据的方法相比，G-RL具有更高精度，且能收敛到更优解。