We apply reinforcement learning (RL) to robotics tasks. One of the drawbacks of traditional RL algorithms has been their poor sample efficiency. One approach to improve the sample efficiency is model-based RL. In our model-based RL algorithm, we learn a model of the environment, essentially its transition dynamics and reward function, use it to generate imaginary trajectories and backpropagate through them to update the policy, exploiting the differentiability of the model. Intuitively, learning more accurate models should lead to better model-based RL performance. Recently, there has been growing interest in developing better deep neural network based dynamics models for physical systems, by utilizing the structure of the underlying physics. We focus on robotic systems undergoing rigid body motion without contacts. We compare two versions of our model-based RL algorithm, one which uses a standard deep neural network based dynamics model and the other which uses a much more accurate, physics-informed neural network based dynamics model. We show that, in model-based RL, model accuracy mainly matters in environments that are sensitive to initial conditions, where numerical errors accumulate fast. In these environments, the physics-informed version of our algorithm achieves significantly better average-return and sample efficiency. In environments that are not sensitive to initial conditions, both versions of our algorithm achieve similar average-return, while the physics-informed version achieves better sample efficiency. We also show that, in challenging environments, physics-informed model-based RL achieves better average-return than state-of-the-art model-free RL algorithms such as Soft Actor-Critic, as it computes the policy-gradient analytically, while the latter estimates it through sampling.

翻译：我们将强化学习（RL）应用于机器人任务。传统RL算法的一个缺点是样本效率低下。提升样本效率的一种方法是基于模型的RL。在我们的基于模型的RL算法中，我们学习环境模型——本质上是其转移动力学和奖励函数，利用该模型生成虚拟轨迹，并通过反向传播更新策略，从而利用模型的可微性。直观上，学习更精确的模型应能提升基于模型RL的性能。近年来，通过利用底层物理结构来开发更优的深度神经网络动力学模型（适用于物理系统）引起了广泛关注。我们聚焦于无接触刚体运动机器人系统。我们比较了两种基于模型RL算法的版本：一种使用标准深度神经网络动力学模型，另一种使用更精确的基于物理信息的神经网络动力学模型。实验表明，在基于模型的RL中，模型精度主要影响对初始条件敏感的环境，其中数值误差会快速累积。在这些环境中，基于物理信息的算法版本在平均回报和样本效率上显著优于标准版本。而在对初始条件不敏感的环境中，两种算法版本虽平均回报相近，但基于物理信息的版本仍展现出更优的样本效率。我们还证明，在复杂环境中，基于物理信息的模型RL比软演员-评论家（Soft Actor-Critic）等最先进的无模型RL算法更优，因为前者能解析计算策略梯度，而后者依赖采样估计。