Legged locomotion has recently achieved remarkable success with the progress of machine learning techniques, especially deep reinforcement learning (RL). Controllers employing neural networks have demonstrated empirical and qualitative robustness against real-world uncertainties, including sensor noise and external perturbations. However, formally investigating the vulnerabilities of these locomotion controllers remains a challenge. This difficulty arises from the requirement to pinpoint vulnerabilities across a long-tailed distribution within a high-dimensional, temporally sequential space. As a first step towards quantitative verification, we propose a computational method that leverages sequential adversarial attacks to identify weaknesses in learned locomotion controllers. Our research demonstrates that, even state-of-the-art robust controllers can fail significantly under well-designed, low-magnitude adversarial sequence. Through experiments in simulation and on the real robot, we validate our approach's effectiveness, and we illustrate how the results it generates can be used to robustify the original policy and offer valuable insights into the safety of these black-box policies. Project page: https://fanshi14.github.io/me/rss24.html

翻译：随着机器学习技术，特别是深度强化学习（RL）的进步，足式运动控制近期取得了显著成功。采用神经网络的控制器在应对现实世界的不确定性（包括传感器噪声和外部扰动）方面，已展现出经验性和定性的鲁棒性。然而，对这些运动控制器的脆弱性进行形式化研究仍然是一个挑战。这一困难源于需要在高度维度、时间序列的空间中，精确定位长尾分布中的脆弱点。作为迈向定量验证的第一步，我们提出一种计算方法，该方法利用序列对抗攻击来识别学习型运动控制器的弱点。我们的研究表明，即使是目前最先进的鲁棒控制器，在精心设计的、低幅度的对抗序列下也可能显著失效。通过在仿真和真实机器人上的实验，我们验证了所提方法的有效性，并说明了如何利用其生成的结果来增强原始策略的鲁棒性，以及为这些黑盒策略的安全性提供有价值的见解。项目页面：https://fanshi14.github.io/me/rss24.html