Batch reinforcement learning (RL) aims at leveraging pre-collected data to find an optimal policy that maximizes the expected total rewards in a dynamic environment. Nearly all existing algorithms rely on the absolutely continuous assumption on the distribution induced by target policies with respect to the data distribution, so that the batch data can be used to calibrate target policies via the change of measure. However, the absolute continuity assumption could be violated in practice (e.g., no-overlap support), especially when the state-action space is large or continuous. In this paper, we propose a new batch RL algorithm without requiring absolute continuity in the setting of an infinite-horizon Markov decision process with continuous states and actions. We call our algorithm STEEL: SingulariTy-awarE rEinforcement Learning. Our algorithm is motivated by a new error analysis on off-policy evaluation, where we use maximum mean discrepancy, together with distributionally robust optimization, to characterize the error of off-policy evaluation caused by the possible singularity and to enable model extrapolation. By leveraging the idea of pessimism and under some mild conditions, we derive a finite-sample regret guarantee for our proposed algorithm without imposing absolute continuity. Compared with existing algorithms, by requiring only minimal data-coverage assumption, STEEL significantly improves the applicability and robustness of batch RL. Extensive simulation studies and one real experiment on personalized pricing demonstrate the superior performance of our method in dealing with possible singularity in batch RL.

翻译：摘要：批强化学习旨在利用预先收集的数据，在动态环境中寻找最大化期望总收益的最优策略。几乎所有现有算法都依赖于目标策略诱导的分布与数据分布之间的绝对连续性假设，从而通过测度变换利用批数据校准目标策略。然而，在实践中（例如支持集无重叠），特别是当状态-动作空间大或连续时，绝对连续性假设可能不成立。本文针对具有连续状态和动作的无限时域马尔可夫决策过程，提出了一种无需绝对连续性假设的新型批强化学习算法，称为STEEL：奇异性感知强化学习。该算法受离策略评估中一种新误差分析的启发，通过使用最大均值差异结合分布鲁棒优化，来刻画由潜在奇异性引起的离策略评估误差，并实现模型外推。通过利用悲观思想并在温和条件下，我们在不施加绝对连续性假设的情况下，为所提算法推导出了有限样本遗憾保证。与现有算法相比，由于仅需最弱的数据覆盖假设，STEEL显著提升了批强化学习的适用性和鲁棒性。大量仿真实验和一项个性化定价真实实验证明了我们的方法在处理批强化学习中潜在奇异性方面的优越性能。