This work aims to tackle a major challenge in offline Inverse Reinforcement Learning (IRL), namely the reward extrapolation error, where the learned reward function may fail to explain the task correctly and misguide the agent in unseen environments due to the intrinsic covariate shift. Leveraging both expert data and lower-quality diverse data, we devise a principled algorithm (namely CLARE) that solves offline IRL efficiently via integrating "conservatism" into a learned reward function and utilizing an estimated dynamics model. Our theoretical analysis provides an upper bound on the return gap between the learned policy and the expert policy, based on which we characterize the impact of covariate shift by examining subtle two-tier tradeoffs between the exploitation (on both expert and diverse data) and exploration (on the estimated dynamics model). We show that CLARE can provably alleviate the reward extrapolation error by striking the right exploitation-exploration balance therein. Extensive experiments corroborate the significant performance gains of CLARE over existing state-of-the-art algorithms on MuJoCo continuous control tasks (especially with a small offline dataset), and the learned reward is highly instructive for further learning.

翻译：本文旨在解决离线逆强化学习（IRL）中的一项主要挑战，即奖励外推误差——由于固有的协变量偏移，学习到的奖励函数可能无法正确解释任务，并在未见环境中误导智能体。我们利用专家数据和低质量多样化数据，设计了一种原则性算法（即CLARE），通过将“保守性”融入学习到的奖励函数并利用估计的动态模型，高效地解决了离线IRL问题。我们的理论分析为学习策略与专家策略之间的回报差距提供了上界，在此基础上，我们通过考察利用（专家数据和多样化数据）与探索（估计的动态模型）之间微妙的双层权衡，刻画了协变量偏移的影响。我们证明，CLARE能够通过在此中达成恰当的利用-探索平衡，在理论上缓解奖励外推误差。大量实验证实，在MuJoCo连续控制任务（尤其是在离线数据集较小的情况下）上，CLARE相较于现有最优算法具有显著的性能提升，且学习到的奖励对进一步学习具有高度指导性。