We explore sim-to-real transfer of deep reinforcement learning controllers for a heavy vehicle with active suspensions designed for traversing rough terrain. While related research primarily focuses on lightweight robots with electric motors and fast actuation, this study uses a forestry vehicle with a complex hydraulic driveline and slow actuation. We simulate the vehicle using multibody dynamics and apply system identification to find an appropriate set of simulation parameters. We then train policies in simulation using various techniques to mitigate the sim-to-real gap, including domain randomization, action delays, and a reward penalty to encourage smooth control. In reality, the policies trained with action delays and a penalty for erratic actions perform at nearly the same level as in simulation. In experiments on level ground, the motion trajectories closely overlap when turning to either side, as well as in a route tracking scenario. When faced with a ramp that requires active use of the suspensions, the simulated and real motions are in close alignment. This shows that the actuator model together with system identification yields a sufficiently accurate model of the actuators. We observe that policies trained without the additional action penalty exhibit fast switching or bang-bang control. These present smooth motions and high performance in simulation but transfer poorly to reality. We find that policies make marginal use of the local height map for perception, showing no indications of look-ahead planning. However, the strong transfer capabilities entail that further development concerning perception and performance can be largely confined to simulation.

翻译：我们探索了深度强化学习控制器从仿真到现实（sim-to-real）的迁移，应用于为穿越崎岖地形而设计的重型车辆主动悬架系统。相关研究主要关注采用电机驱动及快速执行机构的轻量级机器人，而本研究采用了一台具有复杂液压传动系统和慢速执行机构的林业车辆。我们使用多体动力学对车辆进行仿真，并通过系统辨识确定一组合适的仿真参数。随后在仿真中采用多种技术训练策略以缩小仿真与现实的差距，包括域随机化、动作延迟以及用于鼓励平滑控制的奖励惩罚项。实验表明，采用动作延迟和对异常动作施加惩罚项训练的策略在实际环境中表现与仿真几乎相当。在水平地面实验中，车辆转向两侧及路线跟踪场景下的运动轨迹高度重合。当面对需要主动使用悬架的斜坡时，仿真与实际运动高度一致。这表明执行器模型结合系统辨识能够为执行器提供足够精确的模型。我们观察到未添加额外动作惩罚项训练的策略会出现快速切换或"bang-bang"控制现象。这些策略在仿真中呈现平滑运动和高性能表现，但较难迁移至现实环境。分析发现，策略对局部高程图的感知利用程度有限，未表现出前瞻性规划特征。然而，其强大的迁移能力意味着后续关于感知和性能的改进工作可以主要依托仿真环境进行。