Several earlier studies have shown impressive control performance in complex robotic systems by designing the controller using a neural network and training it with model-free reinforcement learning. However, these outstanding controllers with natural motion style and high task performance are developed through extensive reward engineering, which is a highly laborious and time-consuming process of designing numerous reward terms and determining suitable reward coefficients. In this work, we propose a novel reinforcement learning framework for training neural network controllers for complex robotic systems consisting of both rewards and constraints. To let the engineers appropriately reflect their intent to constraints and handle them with minimal computation overhead, two constraint types and an efficient policy optimization algorithm are suggested. The learning framework is applied to train locomotion controllers for several legged robots with different morphology and physical attributes to traverse challenging terrains. Extensive simulation and real-world experiments demonstrate that performant controllers can be trained with significantly less reward engineering, by tuning only a single reward coefficient. Furthermore, a more straightforward and intuitive engineering process can be utilized, thanks to the interpretability and generalizability of constraints. The summary video is available at https://youtu.be/KAlm3yskhvM.

翻译：多项先前研究表明，通过使用神经网络设计控制器并采用无模型强化学习进行训练，能够在复杂机器人系统中实现令人印象深刻的控制性能。然而，这些具有自然运动风格和高任务性能的优秀控制器是通过大量奖励工程开发的，这一过程涉及设计众多奖励项并确定合适的奖励系数，极为费时费力。在本工作中，我们提出了一种新颖的强化学习框架，用于训练复杂机器人系统的神经网络控制器，该框架同时包含奖励和约束。为使工程师能够适当地将其意图反映到约束中，并以最小的计算开销处理这些约束，我们提出了两种约束类型和一种高效策略优化算法。该学习框架被应用于训练多种具有不同形态和物理属性的足式机器人的运动控制器，以使其穿越具有挑战性的地形。大量仿真和真实世界实验表明，通过仅调整单个奖励系数，即可在显著减少奖励工程的情况下训练出高性能控制器。此外，由于约束的可解释性和泛化性，可以采用更直接和直观的工程过程。总结视频见 https://youtu.be/KAlm3yskhvM。