The introduction of the generative adversarial imitation learning (GAIL) algorithm has spurred the development of scalable imitation learning approaches using deep neural networks. Many of the algorithms that followed used a similar procedure, combining on-policy actor-critic algorithms with inverse reinforcement learning. More recently there have been an even larger breadth of approaches, most of which use off-policy algorithms. However, with the breadth of algorithms, everything from datasets to base reinforcement learning algorithms to evaluation settings can vary, making it difficult to fairly compare them. In this work we re-implement 6 different IL algorithms, updating 3 of them to be off-policy, base them on a common off-policy algorithm (SAC), and evaluate them on a widely-used expert trajectory dataset (D4RL) for the most common benchmark (MuJoCo). After giving all algorithms the same hyperparameter optimisation budget, we compare their results for a range of expert trajectories. In summary, GAIL, with all of its improvements, consistently performs well across a range of sample sizes, AdRIL is a simple contender that performs well with one important hyperparameter to tune, and behavioural cloning remains a strong baseline when data is more plentiful.

翻译：生成对抗模仿学习（GAIL）算法的提出推动了基于深度神经网络的可扩展模仿学习方法的发展。随后许多算法采用类似流程，将基于策略的演员-评论家算法与逆强化学习相结合。近期涌现出更广泛的方法，其中多数采用离策略算法。然而，由于算法种类繁多，从数据集到基础强化学习算法再到评估设置均存在差异，导致难以进行公平比较。本研究重新实现了6种不同的模仿学习算法，将其中3种升级为离策略形式，统一基于通用离策略算法SAC，并在最常用基准测试MuJoCo的广泛使用专家轨迹数据集D4RL上进行评估。在赋予所有算法相同的超参数优化预算后，我们比较它们在一系列专家轨迹下的表现。总结而言，具备全部改进特性的GAIL在不同样本规模下均表现稳定，AdRIL作为简单候选算法只需调整一项关键超参数即可获得良好性能，而行为克隆在数据更充足时仍保持强劲基线水平。