Imitation learning addresses the challenge of learning by observing an expert's demonstrations without access to reward signals from environments. Most existing imitation learning methods that do not require interacting with environments either model the expert distribution as the conditional probability p(a|s) (e.g., behavioral cloning, BC) or the joint probability p(s, a) (e.g., implicit behavioral cloning). Despite its simplicity, modeling the conditional probability with BC usually struggles with generalization. While modeling the joint probability can lead to improved generalization performance, the inference procedure can be time-consuming and it often suffers from manifold overfitting. This work proposes an imitation learning framework that benefits from modeling both the conditional and joint probability of the expert distribution. Our proposed diffusion model-augmented behavioral cloning (DBC) employs a diffusion model trained to model expert behaviors and learns a policy to optimize both the BC loss (conditional) and our proposed diffusion model loss (joint). DBC outperforms baselines in various continuous control tasks in navigation, robot arm manipulation, dexterous manipulation, and locomotion. We design additional experiments to verify the limitations of modeling either the conditional probability or the joint probability of the expert distribution as well as compare different generative models.

翻译：模仿学习旨在通过观察专家的演示来学习，而无需从环境中获取奖励信号。大多数无需与环境交互的现有模仿学习方法，要么将专家分布建模为条件概率p(a|s)（例如行为克隆，BC），要么建模为联合概率p(s, a)（例如隐式行为克隆）。尽管BC建模条件概率的方法简单，但通常难以实现泛化。而建模联合概率虽能提升泛化性能，但推理过程耗时且易出现流形过拟合问题。本文提出了一种通过同时建模专家分布的条件概率与联合概率来获益的模仿学习框架。我们提出的扩散模型增强行为克隆（DBC）采用扩散模型来学习专家行为，并训练策略以同时优化BC损失（条件概率）和本文提出的扩散模型损失（联合概率）。在导航、机械臂操作、灵巧操控及运动控制等多种连续控制任务中，DBC的性能优于基线方法。我们还设计额外实验验证了仅建模条件概率或联合概率的局限性，并比较了不同生成模型的性能。