Continuous-time reinforcement learning tasks commonly use discrete steps of fixed cycle times for actions. As practitioners need to choose the action-cycle time for a given task, a significant concern is whether the hyper-parameters of the learning algorithm need to be re-tuned for each choice of the cycle time, which is prohibitive for real-world robotics. In this work, we investigate the widely-used baseline hyper-parameter values of two policy gradient algorithms -- PPO and SAC -- across different cycle times. Using a benchmark task where the baseline hyper-parameters of both algorithms were shown to work well, we reveal that when a cycle time different than the task default is chosen, PPO with baseline hyper-parameters fails to learn. Moreover, both PPO and SAC with their baseline hyper-parameters perform substantially worse than their tuned values for each cycle time. We propose novel approaches for setting these hyper-parameters based on the cycle time. In our experiments on simulated and real-world robotic tasks, the proposed approaches performed at least as well as the baseline hyper-parameters, with significantly better performance for most choices of the cycle time, and did not result in learning failure for any cycle time. Hyper-parameter tuning still remains a significant barrier for real-world robotics, as our approaches require some initial tuning on a new task, even though it is negligible compared to an extensive tuning for each cycle time. Our approach requires no additional tuning after the cycle time is changed for a given task and is a step toward avoiding extensive and costly hyper-parameter tuning for real-world policy optimization.

翻译：连续时间强化学习任务通常采用固定周期时间的离散动作步长。由于实践者需要为给定任务选择动作周期时间，一个关键问题在于：学习算法的超参数是否需要针对每个周期时间选择重新调整，这对实际机器人应用而言代价过高。本研究探究了两种策略梯度算法（PPO和SAC）在不同周期时间下的基线超参数值。通过使用一个基线超参数已被证明有效的基准任务，我们发现：当选择的周期时间与任务默认值不同时，采用基线超参数的PPO算法将无法学习。此外，两种算法使用基线超参数的性能均显著低于针对每个周期时间调优后的参数。我们提出了基于周期时间设置这些超参数的新方法。在仿真与实际机器人任务的实验中，所提方法的表现至少不逊于基线超参数，且在大多数周期时间选择下性能显著更优，且未在任何周期时间下导致学习失败。超参数调节仍是实际机器人应用的主要障碍——尽管相比为每个周期时间进行广泛调优而言微不足道，我们的方法仍需在新任务上完成初始调优。本方法在给定任务中改变周期时间后无需额外调优，为减少实际策略优化中昂贵且广泛的超参数调优迈出了重要一步。