Traditional reinforcement learning (RL) methods typically employ a fixed control loop, where each cycle corresponds to an action. This rigidity poses challenges in practical applications, as the optimal control frequency is task-dependent. A suboptimal choice can lead to high computational demands and reduced exploration efficiency. Variable Time Step Reinforcement Learning (VTS-RL) addresses these issues by using adaptive frequencies for the control loop, executing actions only when necessary. This approach, rooted in reactive programming principles, reduces computational load and extends the action space by including action durations. However, VTS-RL's implementation is often complicated by the need to tune multiple hyperparameters that govern exploration in the multi-objective action-duration space (i.e., balancing task performance and number of time steps to achieve a goal). To overcome these challenges, we introduce the Multi-Objective Soft Elastic Actor-Critic (MOSEAC) method. This method features an adaptive reward scheme that adjusts hyperparameters based on observed trends in task rewards during training. This scheme reduces the complexity of hyperparameter tuning, requiring a single hyperparameter to guide exploration, thereby simplifying the learning process and lowering deployment costs. We validate the MOSEAC method through simulations in a Newtonian kinematics environment, demonstrating high task and training performance with fewer time steps, ultimately lowering energy consumption. This validation shows that MOSEAC streamlines RL algorithm deployment by automatically tuning the agent control loop frequency using a single parameter. Its principles can be applied to enhance any RL algorithm, making it a versatile solution for various applications.

翻译：传统强化学习方法通常采用固定控制循环，每个周期对应一个动作。这种刚性在实际应用中带来挑战，因为最优控制频率与任务相关。次优选择可能导致高计算需求和探索效率降低。可变时间步长强化学习通过采用自适应频率的控制循环来解决这些问题，仅在必要时执行动作。该方法基于反应式编程原理，通过包含动作持续时间来减少计算负载并扩展动作空间。然而，VTS-RL的实现往往因需要调整多个超参数而变得复杂，这些超参数控制着多目标动作-持续时间空间中的探索（即平衡任务性能与达成目标所需的时间步数）。为克服这些挑战，我们提出了多目标软弹性演员-评论家方法。该方法采用自适应奖励机制，根据训练期间观察到的任务奖励趋势调整超参数。该机制降低了超参数调优的复杂性，仅需单个超参数来引导探索，从而简化学习过程并降低部署成本。我们在牛顿运动学环境中通过仿真验证了MOSEAC方法，证明其能以更少的时间步数实现较高的任务和训练性能，最终降低能耗。该验证表明，MOSEAC通过使用单一参数自动调整智能体控制循环频率，简化了强化学习算法的部署。其原理可应用于增强任何强化学习算法，使其成为适用于各种应用的通用解决方案。