In this paper, we define, evaluate, and improve the ``relay-generalization'' performance of reinforcement learning (RL) agents on the out-of-distribution ``controllable'' states. Ideally, an RL agent that generally masters a task should reach its goal starting from any controllable state of the environment instead of memorizing a small set of trajectories. For example, a self-driving system should be able to take over the control from humans in the middle of driving and must continue to drive the car safely. To practically evaluate this type of generalization, we start the test agent from the middle of other independently well-trained \emph{stranger} agents' trajectories. With extensive experimental evaluation, we show the prevalence of \emph{generalization failure} on controllable states from stranger agents. For example, in the Humanoid environment, we observed that a well-trained Proximal Policy Optimization (PPO) agent, with only 3.9\% failure rate during regular testing, failed on 81.6\% of the states generated by well-trained stranger PPO agents. To improve "relay generalization," we propose a novel method called Self-Trajectory Augmentation (STA), which will reset the environment to the agent's old states according to the Q function during training. After applying STA to the Soft Actor Critic's (SAC) training procedure, we reduced the failure rate of SAC under relay-evaluation by more than three times in most settings without impacting agent performance and increasing the needed number of environment interactions. Our code is available at https://github.com/lan-lc/STA.

翻译：本文定义、评估并改进了强化学习（RL）智能体在分布外“可控”状态上的“接力泛化”性能。理想情况下，一个完全掌握任务的RL智能体应从环境的任意可控状态起始达成目标，而非仅记忆少量轨迹。例如，自动驾驶系统需能在行驶中途从人类驾驶员手中接管控制权，并持续安全驾驶。为实际评估此类泛化能力，我们让测试智能体从其他独立训练完成的“陌生人”智能体轨迹中途出发。通过大量实验评估，我们发现智能体在陌生人轨迹的可控状态上普遍存在“泛化失败”现象。例如，在Humanoid环境中，一个训练良好的近端策略优化（PPO）智能体常规测试失败率仅3.9%，但在其他训练良好的陌生人PPO智能体所生成的状态上，其失败率高达81.6%。为提升“接力泛化”，我们提出一种名为自轨迹增强（STA）的新方法，该方法根据Q函数在训练过程中将环境重置至智能体的历史状态。将STA应用于软演员-评论家（SAC）训练流程后，在多数设定下SAC在接力评估中的失败率降低三倍以上，且不影响智能体性能或增加所需的环境交互次数。我们的代码已开源：https://github.com/lan-lc/STA。