Simulation-to-reality (sim-to-real) transfer is a fundamental problem for robot learning. Domain Randomization, which adds randomization during training, is a powerful technique that effectively addresses the sim-to-real gap. However, the noise in observations makes learning significantly harder. Recently, studies have shown that employing a teacher-student learning paradigm can accelerate training in randomized environments. Learned with privileged information, a teacher agent can instruct the student agent to operate in noisy environments. However, this approach is often not sample efficient as the experience collected by the teacher is discarded completely when training the student, wasting information revealed by the environment. In this work, we extend the teacher-student learning paradigm by proposing a sample efficient learning framework termed Learn to Teach (L2T) that recycles experience collected by the teacher agent. We observe that the dynamics of the environments for both agents remain unchanged, and the state space of the teacher is coupled with the observation space of the student. We show that a single-loop algorithm can train both the teacher and student agents under both Reinforcement Learning and Inverse Reinforcement Learning contexts. We implement variants of our methods, conduct experiments on the MuJoCo benchmark, and apply our methods to the Cassie robot locomotion problem. Extensive experiments show that our method achieves competitive performance while only requiring environmental interaction with the teacher.

翻译：仿真到现实（sim-to-real）迁移是机器人学习中的一个基本问题。域随机化（Domain Randomization）作为一种在训练过程中添加随机化的强大技术，能够有效弥合仿真与现实之间的差距。然而，观测中的噪声使学习过程显著变得更加困难。近年来，研究表明，采用师生学习范式可以加速在随机化环境中的训练。通过利用特权信息进行学习，教师智能体可以指导学生智能体在噪声环境中运行。然而，这种方法通常样本效率不高，因为教师收集的经验在训练学生时被完全丢弃，浪费了环境所揭示的信息。在本工作中，我们通过提出一种名为“学会教授”（Learn to Teach, L2T）的样本高效学习框架，扩展了师生学习范式，该框架能够回收教师智能体收集的经验。我们观察到，两个智能体的环境动态保持不变，且教师的状态空间与学生的观测空间是耦合的。我们证明，一个单循环算法可以在强化学习和逆强化学习背景下同时训练教师和学生智能体。我们实现了多种方法变体，在MuJoCo基准测试上进行了实验，并将我们的方法应用于Cassie机器人运动问题。大量实验表明，我们的方法在仅需与环境进行教师交互的情况下，即可实现具有竞争力的性能。