Imitation of expert behaviour is a highly desirable and safe approach to the problem of sequential decision making. We provide an easy-to-implement, novel algorithm for imitation learning under a strict data paradigm, in which the agent must learn solely from data collected a priori. This paradigm allows our algorithm to be used for environments in which safety or cost are of critical concern. Our algorithm requires no additional hyper-parameter tuning beyond any standard batch reinforcement learning (RL) algorithm, making it an ideal baseline for such data-strict regimes. Furthermore, we provide formal sample complexity guarantees for the algorithm in finite Markov Decision Problems. In doing so, we formally demonstrate an unproven claim from Kearns & Singh (1998). On the empirical side, our contribution is twofold. First, we develop a practical, robust and principled evaluation protocol for offline RL methods, making use of only the dataset provided for model selection. This stands in contrast to the vast majority of previous works in offline RL, which tune hyperparameters on the evaluation environment, limiting the practical applicability when deployed in new, cost-critical environments. As such, we establish precedent for the development and fair evaluation of offline RL algorithms. Second, we evaluate our own algorithm on challenging continuous control benchmarks, demonstrating its practical applicability and competitiveness with state-of-the-art performance, despite being a simpler algorithm.

翻译：模仿专家行为是解决序贯决策问题的一种理想且安全的方法。针对智能体必须仅从预先收集的数据中学习的严格数据范式，我们提出了一种易于实现的新型模仿学习算法。该范式使得我们的算法可应用于安全或成本至关重要的环境。与任何标准批量强化学习算法相比，该算法无需额外超参数调优，因此成为此类数据严格场景下的理想基线方法。此外，我们为有限马尔可夫决策问题中的算法提供了正式的样本复杂度保证，从而严格证明了Kearns & Singh (1998)中未经验证的主张。在实证方面，我们的贡献包含两点。首先，我们为离线强化学习方法建立了一套实用、稳健且规范化的评估协议，仅利用数据集进行模型选择。这与大多数离线强化学习先前工作形成鲜明对比——后者通过在评估环境中调整超参数，限制了其在成本关键型新环境部署中的实践可行性。基于此，我们为离线强化学习算法的开发与公平评估确立了先例。其次，我们在具有挑战性的连续控制基准上评估了所提算法，尽管其结构更简洁，仍展现出与现有最优方法相匹敌的实践适用性与竞争力。