In Reinforcement Learning, agents learn policies by exploring and interacting with the environment. Due to the curse of dimensionality, learning policies that map high-dimensional sensory input to motor output is particularly challenging. During training, state of the art methods (SAC, PPO, etc.) explore the environment by perturbing the actuation with independent Gaussian noise. While this unstructured exploration has proven successful in numerous tasks, it can be suboptimal for overactuated systems. When multiple actuators, such as motors or muscles, drive behavior, uncorrelated perturbations risk diminishing each other's effect, or modifying the behavior in a task-irrelevant way. While solutions to introduce time correlation across action perturbations exist, introducing correlation across actuators has been largely ignored. Here, we propose LATent TIme-Correlated Exploration (Lattice), a method to inject temporally-correlated noise into the latent state of the policy network, which can be seamlessly integrated with on- and off-policy algorithms. We demonstrate that the noisy actions generated by perturbing the network's activations can be modeled as a multivariate Gaussian distribution with a full covariance matrix. In the PyBullet locomotion tasks, Lattice-SAC achieves state of the art results, and reaches 18% higher reward than unstructured exploration in the Humanoid environment. In the musculoskeletal control environments of MyoSuite, Lattice-PPO achieves higher reward in most reaching and object manipulation tasks, while also finding more energy-efficient policies with reductions of 20-60%. Overall, we demonstrate the effectiveness of structured action noise in time and actuator space for complex motor control tasks. The code is available at: https://github.com/amathislab/lattice.

翻译：在强化学习中，智能体通过探索和与环境交互来学习策略。由于维度灾难，学习将高维感官输入映射到运动输出的策略尤为困难。在训练过程中，最先进的方法（如SAC、PPO等）通过向执行机构添加独立高斯噪声来探索环境。虽然这种非结构化探索在众多任务中已被证明有效，但对于过驱动系统而言，它可能并非最优。当多个执行器（如电机或肌肉）驱动行为时，不相关的扰动可能会相互抵消影响，或以与任务无关的方式改变行为。尽管已有解决方案引入跨动作扰动的时间相关性，但引入执行器间的相关性在很大程度上被忽视了。为此，我们提出潜在时间相关探索（Lattice）方法，该方法将时间相关噪声注入策略网络的潜在状态，并能无缝集成到同策略和离策略算法中。我们证明：通过扰动网络激活产生的噪声动作可建模为具有全协方差矩阵的多元高斯分布。在PyBullet运动任务中，Lattice-SAC取得了最先进的结果，并在Humanoid环境中比非结构化探索获得高达18%的奖励。在MyoSuite的肌肉骨骼控制环境中，Lattice-PPO在大多数抓取和物体操作任务中获得了更高的奖励，同时发现了更节能的策略，能耗降低20-60%。总体而言，我们证明了在时间和执行器空间中的结构化动作噪声对复杂运动控制任务的有效性。代码见：https://github.com/amathislab/lattice。