Offline inverse reinforcement learning (Offline IRL) aims to recover the structure of rewards and environment dynamics that underlie observed actions in a fixed, finite set of demonstrations from an expert agent. Accurate models of expertise in executing a task has applications in safety-sensitive applications such as clinical decision making and autonomous driving. However, the structure of an expert's preferences implicit in observed actions is closely linked to the expert's model of the environment dynamics (i.e. the ``world''). Thus, inaccurate models of the world obtained from finite data with limited coverage could compound inaccuracy in estimated rewards. To address this issue, we propose a bi-level optimization formulation of the estimation task wherein the upper level is likelihood maximization based upon a conservative model of the expert's policy (lower level). The policy model is conservative in that it maximizes reward subject to a penalty that is increasing in the uncertainty of the estimated model of the world. We propose a new algorithmic framework to solve the bi-level optimization problem formulation and provide statistical and computational guarantees of performance for the associated reward estimator. Finally, we demonstrate that the proposed algorithm outperforms the state-of-the-art offline IRL and imitation learning benchmarks by a large margin, over the continuous control tasks in MuJoCo and different datasets in the D4RL benchmark.

翻译：离线逆向强化学习旨在从专家代理固定且有限的演示集合中，恢复驱动观察动作的奖励与环境动态的结构。对任务执行专长的精确建模在临床决策和自动驾驶等安全敏感应用中具有重要价值。然而，观察动作中隐含的专家偏好结构与专家对环境动态（即"世界"）的模型密切相关。因此，从覆盖范围有限的有限数据中获取的不准确世界模型可能加剧估计奖励的误差。为解决此问题，我们提出一种双层优化估计框架，其中上层基于专家策略的保守模型（下层）进行似然最大化。该策略模型具有保守性，其通过最大化奖励并施加随估计世界模型不确定性增加的惩罚项实现。我们提出一种新的算法框架来求解该双层优化问题，并为相关奖励估计器提供统计与计算性能保证。最后，我们证明所提算法在MuJoCo连续控制任务和D4RL基准的不同数据集上，以显著优势优于现有最优的离线逆向强化学习及模仿学习基准方法。