In Reinforcement Learning, agents learn policies by exploring and interacting with the environment. Due to the curse of dimensionality, learning policies that map high-dimensional sensory input to motor output is particularly challenging. During training, state of the art methods (SAC, PPO, etc.) explore the environment by perturbing the actuation with independent Gaussian noise. While this unstructured exploration has proven successful in numerous tasks, it ought to be suboptimal for overactuated systems. When multiple actuators, such as motors or muscles, drive behavior, uncorrelated perturbations risk diminishing each other's effect, or modifying the behavior in a task-irrelevant way. While solutions to introduce time correlation across action perturbations exist, introducing correlation across actuators has been largely ignored. Here, we propose LATent TIme-Correlated Exploration (Lattice), a method to inject temporally-correlated noise into the latent state of the policy network, which can be seamlessly integrated with on- and off-policy algorithms. We demonstrate that the noisy actions generated by perturbing the network's activations can be modeled as a multivariate Gaussian distribution with a full covariance matrix. In the PyBullet locomotion tasks, Lattice-SAC achieves state of the art results, and reaches 18% higher reward than unstructured exploration in the Humanoid environment. In the musculoskeletal control environments of MyoSuite, Lattice-PPO achieves higher reward in most reaching and object manipulation tasks, while also finding more energy-efficient policies with reductions of 20-60%. Overall, we demonstrate the effectiveness of structured action noise in time and actuator space for complex motor control tasks.

翻译：在强化学习中，智能体通过与环境的交互和探索来学习策略。由于维度灾难，学习将高维感官输入映射到运动输出的策略尤为困难。在训练过程中，最先进的方法（如SAC、PPO等）通过向执行动作添加独立的高斯噪声来探索环境。尽管这种非结构化探索在许多任务中取得了成功，但对于多驱动系统而言，它可能是次优的。当多个执行器（如电机或肌肉）驱动行为时，不相关的扰动可能会相互抵消效果，或以与任务无关的方式改变行为。虽然已有解决方案引入动作扰动的时间相关性，但引入执行器间的相关性在很大程度上被忽视了。在此，我们提出LATent TIme-Correlated Exploration（Lattice）方法，该方法将时间相关的噪声注入策略网络的潜在状态，并可无缝集成到在线和离线策略算法中。我们证明，通过扰动网络激活生成的带噪动作可建模为具有全协方差矩阵的多元高斯分布。在PyBullet运动任务中，Lattice-SAC取得了最先进的结果，在Humanoid环境中比非结构化探索的奖励高18%。在MyoSuite的肌肉骨骼控制环境中，Lattice-PPO在大多数伸手和物体操作任务中获得了更高的奖励，同时找到了更节能的策略，能耗降低20-60%。总体而言，我们展示了在时间和执行器空间中的结构化动作噪声在复杂运动控制任务中的有效性。