When a gait of a bipedal robot is developed using deep reinforcement learning, reference trajectories may or may not be used. Each approach has its advantages and disadvantages, and the choice of method is up to the control developer. This paper investigates the effect of reference trajectories on locomotion learning and the resulting gaits. We implemented three gaits of a full-order anthropomorphic robot model with different reward imitation ratios, provided sim-to-sim control policy transfer, and compared the gaits in terms of robustness and energy efficiency. In addition, we conducted a qualitative analysis of the gaits by interviewing people, since our task was to create an appealing and natural gait for a humanoid robot. According to the results of the experiments, the most successful approach was the one in which the average value of rewards for imitation and adherence to command velocity per episode remained balanced throughout the training. The gait obtained with this method retains naturalness (median of 3.6 according to the user study) compared to the gait trained with imitation only (median of 4.0), while remaining robust close to the gait trained without reference trajectories.

翻译：当使用深度强化学习开发双足机器人步态时，参考轨迹既可能被使用也可能不被使用。每种方法都有其优缺点，方法的选择取决于控制开发人员。本文研究了参考轨迹对运动学习及由此产生的步态的影响。我们实现了一个全阶拟人机器人模型的三种步态，并采用了不同的奖励模仿比例，提供了从仿真到仿真的控制策略迁移，并从鲁棒性和能量效率方面对这些步态进行了比较。此外，我们还通过访谈进行了步态的定性分析，因为我们的任务是为人形机器人创建一种自然且具有吸引力的步态。实验结果表明，最成功的方法是每回合模仿奖励与指令速度遵循奖励的平均值在整个训练过程中保持平衡的方法。通过该方法获得的步态保持了自然性（根据用户研究，中位数为3.6），与仅通过模仿训练的步态（中位数为4.0）相比，同时其鲁棒性接近于无参考轨迹训练的步态。