In this paper, we focus on single-demonstration imitation learning (IL), a practical approach for real-world applications where acquiring multiple expert demonstrations is costly or infeasible and the ground truth reward function is not available. In contrast to typical IL settings with multiple demonstrations, single-demonstration IL involves an agent having access to only one expert trajectory. We highlight the issue of sparse reward signals in this setting and propose to mitigate this issue through our proposed Transition Discriminator-based IL (TDIL) method. TDIL is an IRL method designed to address reward sparsity by introducing a denser surrogate reward function that considers environmental dynamics. This surrogate reward function encourages the agent to navigate towards states that are proximal to expert states. In practice, TDIL trains a transition discriminator to differentiate between valid and non-valid transitions in a given environment to compute the surrogate rewards. The experiments demonstrate that TDIL outperforms existing IL approaches and achieves expert-level performance in the single-demonstration IL setting across five widely adopted MuJoCo benchmarks as well as the "Adroit Door" robotic environment.

翻译：本文聚焦于单演示模仿学习，这是一种在实际应用中具有实用价值的方法，尤其适用于获取多专家演示成本高昂或不可行、且无法获得真实奖励函数的场景。与通常使用多演示的模仿学习设置不同，单演示模仿学习中智能体仅能访问一条专家轨迹。我们指出了此设置下奖励信号稀疏的问题，并提出通过我们设计的基于转移判别器的模仿学习方法来解决这一问题。TDIL是一种逆向强化学习方法，旨在通过引入一个考虑环境动态的、更密集的替代奖励函数来应对奖励稀疏性。该替代奖励函数激励智能体导航至接近专家状态的状态。在实践中，TDIL通过训练一个转移判别器来区分给定环境中的有效与无效转移，从而计算替代奖励。实验表明，在五个广泛采用的MuJoCo基准测试以及"Adroit Door"机器人环境中，TDIL在单演示模仿学习设置下超越了现有模仿学习方法，并达到了专家级性能。