Deploying controllers trained with Reinforcement Learning (RL) on real robots can be challenging: RL relies on agents' policies being modeled as Markov Decision Processes (MDPs), which assume an inherently discrete passage of time. The use of MDPs results in that nearly all RL-based control systems employ a fixed-rate control strategy with a period (or time step) typically chosen based on the developer's experience or specific characteristics of the application environment. Unfortunately, the system should be controlled at the highest, worst-case frequency to ensure stability, which can demand significant computational and energy resources and hinder the deployability of the controller on onboard hardware. Adhering to the principles of reactive programming, we surmise that applying control actions only when necessary enables the use of simpler hardware and helps reduce energy consumption. We challenge the fixed frequency assumption by proposing a variant of RL with variable control rate. In this approach, the policy decides the action the agent should take as well as the duration of the time step associated with that action. In our new setting, we expand Soft Actor-Critic (SAC) to compute the optimal policy with a variable control rate, introducing the Soft Elastic Actor-Critic (SEAC) algorithm. We show the efficacy of SEAC through a proof-of-concept simulation driving an agent with Newtonian kinematics. Our experiments show higher average returns, shorter task completion times, and reduced computational resources when compared to fixed rate policies.

翻译：在真实机器人上部署基于强化学习训练的控制器具有挑战性：强化学习依赖的智能体策略被建模为马尔可夫决策过程，该过程假设时间具有内在的离散性。马尔可夫决策过程的使用导致几乎所有基于强化学习的控制系统都采用固定周期的控制策略，其控制周期（或时间步长）通常根据开发者的经验或应用环境的特定特征确定。不幸的是，为确保系统稳定性，必须以最高最坏情况频率进行控制，这将消耗大量计算和能源资源，并阻碍控制器在机载硬件上的可部署性。遵循反应式编程原则，我们推测仅在必要时施加控制动作能够使用更简单的硬件并有助于降低能耗。我们通过提出一种可变控制率的强化学习变体来挑战固定频率假设。在该方法中，策略不仅决定智能体应执行的动作，还决定该动作对应的时间步长持续时间。在新框架下，我们将柔性演员-评论家算法扩展至可变控制率场景以计算最优策略，由此提出软弹性演员-评论家算法。通过驱动基于牛顿运动学智能体的概念验证仿真，我们展示了SEAC算法的有效性。实验表明，与固定频率策略相比，本方法可获得更高的平均回报、更短的任务完成时间，并减少计算资源消耗。