The Teacher-Student Framework (TSF) is a reinforcement learning setting where a teacher agent guards the training of a student agent by intervening and providing online demonstrations. Assuming optimal, the teacher policy has the perfect timing and capability to intervene in the learning process of the student agent, providing safety guarantee and exploration guidance. Nevertheless, in many real-world settings it is expensive or even impossible to obtain a well-performing teacher policy. In this work, we relax the assumption of a well-performing teacher and develop a new method that can incorporate arbitrary teacher policies with modest or inferior performance. We instantiate an Off-Policy Reinforcement Learning algorithm, termed Teacher-Student Shared Control (TS2C), which incorporates teacher intervention based on trajectory-based value estimation. Theoretical analysis validates that the proposed TS2C algorithm attains efficient exploration and substantial safety guarantee without being affected by the teacher's own performance. Experiments on various continuous control tasks show that our method can exploit teacher policies at different performance levels while maintaining a low training cost. Moreover, the student policy surpasses the imperfect teacher policy in terms of higher accumulated reward in held-out testing environments. Code is available at https://metadriverse.github.io/TS2C.

翻译：教师-学生框架是一种强化学习设置，其中教师智能体通过干预并提供在线演示来保护学生智能体的训练。假设教师策略最优，则其具备完美时机和能力来干预学生智能体的学习过程，从而提供安全保障和探索引导。然而，在许多现实场景中，获得表现良好的教师策略成本高昂甚至不可能。本文放宽了教师策略表现良好的假设，开发了一种能够整合任意性能平庸或低下教师策略的新方法。我们实例化了一种离策略强化学习算法，称为教师-学生共享控制（TS2C），该算法基于轨迹价值估计引入教师干预。理论分析验证了所提出的TS2C算法能够实现高效探索和充分安全保障，且不受教师自身性能影响。在各种连续控制任务上的实验表明，我们的方法可以在保持低训练成本的同时，利用不同性能水平的教师策略。此外，在保留测试环境中，学生策略在累积奖励方面超越了不完美的教师策略。代码可在https://metadriverse.github.io/TS2C获取。