Human-AI shared control allows human to interact and collaborate with AI to accomplish control tasks in complex environments. Previous Reinforcement Learning (RL) methods attempt the goal-conditioned design to achieve human-controllable policies at the cost of redesigning the reward function and training paradigm. Inspired by the neuroscience approach to investigate the motor cortex in primates, we develop a simple yet effective frequency-based approach called \textit{Policy Dissection} to align the intermediate representation of the learned neural controller with the kinematic attributes of the agent behavior. Without modifying the neural controller or retraining the model, the proposed approach can convert a given RL-trained policy into a human-interactive policy. We evaluate the proposed approach on the RL tasks of autonomous driving and locomotion. The experiments show that human-AI shared control achieved by Policy Dissection in driving task can substantially improve the performance and safety in unseen traffic scenes. With human in the loop, the locomotion robots also exhibit versatile controllable motion skills even though they are only trained to move forward. Our results suggest the promising direction of implementing human-AI shared autonomy through interpreting the learned representation of the autonomous agents. Demo video and code will be made available at https://metadriverse.github.io/policydissect.

翻译：人机共享控制使人类能够与人工智能交互协作，在复杂环境中完成控制任务。以往的强化学习方法尝试通过目标条件设计来实现人类可控策略，但这需要重新设计奖励函数和训练范式。受神经科学中研究灵长类动物运动皮层方法的启发，我们提出了一种简单而有效的基于频率的方法——"策略剖析"（Policy Dissection），将所学神经控制器的中间表征与智能体行为的运动学属性对齐。该方法无需修改神经控制器或重新训练模型，即可将给定的强化学习训练策略转化为人类可交互策略。我们在自动驾驶和移动控制等强化学习任务上评估了该方法。实验表明，通过策略剖析实现的驾驶任务人机共享控制，能在未见过的交通场景中显著提升性能与安全性。在人类参与下，即使移动机器人仅接受过前进训练，也能展现出多样化的可控运动技能。我们的研究结果表明，通过解读自主智能体的学习表征来实现人机共享自主性是一条富有前景的路径。演示视频和代码将在 https://metadriverse.github.io/policydissect 提供。