Imitation learning (IL) is a paradigm for learning sequential decision making policies from experts, leveraging offline demonstrations, interactive annotations, or both. Recent advances show that when annotation cost is tallied per trajectory, Behavior Cloning (BC) which relies solely on offline demonstrations cannot be improved in general, leaving limited conditions for interactive methods such as DAgger to help. We revisit this conclusion and prove that when the annotation cost is measured per state, algorithms using interactive annotations can provably outperform BC. Specifically: (1) we show that Stagger, a one sample per round variant of DAgger, provably beats BC under low recovery cost settings; (2) we initiate the study of hybrid IL where the agent learns from offline demonstrations and interactive annotations. We propose Warm Stagger whose learning guarantee is not much worse than using either data source alone. Furthermore, motivated by compounding error and cold start problem in imitation learning practice, we give an MDP example in which Warm Stagger has significant better annotation cost; (3) experiments on MuJoCo continuous control tasks confirm that, with modest cost ratio between interactive and offline annotations, interactive and hybrid approaches consistently outperform BC. To the best of our knowledge, our work is the first to highlight the benefit of state wise interactive annotation and hybrid feedback in imitation learning.

翻译：模仿学习（IL）是一种从专家处学习序列决策策略的范式，其利用离线演示、交互式标注或两者结合。近期研究表明，当按轨迹计算标注成本时，仅依赖离线演示的行为克隆（BC）在一般情况下无法被改进，这使得DAgger等交互式方法的适用条件受限。我们重新审视这一结论并证明，当按状态衡量标注成本时，采用交互式标注的算法可证明优于BC。具体而言：（1）我们证明Stagger（DAgger的每轮单样本变体）在低恢复成本设置下可证明优于BC；（2）我们开创性地研究混合IL，其中智能体同时从离线演示和交互式标注中学习。我们提出Warm Stagger，其学习保证不逊于单独使用任一数据源。进一步地，针对模仿学习实践中存在的复合误差与冷启动问题，我们给出一个MDP示例，其中Warm Stagger具有显著更优的标注成本；（3）在MuJoCo连续控制任务上的实验证实，当交互式与离线标注成本比率适中时，交互式与混合方法始终优于BC。据我们所知，本研究首次系统揭示了状态级交互式标注与混合反馈在模仿学习中的优势。