Batch reinforcement learning (RL) aims at finding an optimal policy in a dynamic environment in order to maximize the expected total rewards by leveraging pre-collected data. A fundamental challenge behind this task is the distributional mismatch between the batch data generating process and the distribution induced by target policies. Nearly all existing algorithms rely on the absolutely continuous assumption on the distribution induced by target policies with respect to the data distribution so that the batch data can be used to calibrate target policies via the change of measure. However, the absolute continuity assumption could be violated in practice, especially when the state-action space is large or continuous. In this paper, we propose a new batch RL algorithm without requiring absolute continuity in the setting of an infinite-horizon Markov decision process with continuous states and actions. We call our algorithm STEEL: SingulariTy-awarE rEinforcement Learning. Our algorithm is motivated by a new error analysis on off-policy evaluation, where we use maximum mean discrepancy, together with distributionally robust optimization, to characterize the error of off-policy evaluation caused by the possible singularity and to enable the power of model extrapolation. By leveraging the idea of pessimism and under some mild conditions, we derive a finite-sample regret guarantee for our proposed algorithm without imposing absolute continuity. Compared with existing algorithms, STEEL only requires some minimal data-coverage assumption and thus greatly enhances the applicability and robustness of batch RL. Extensive simulation studies and one real experiment on personalized pricing demonstrate the superior performance of our method when facing possible singularity in batch RL.

翻译：批量强化学习旨在通过利用预先收集的数据，在动态环境中找到最优策略，以最大化期望总回报。该任务的一个基本挑战是批量数据生成过程与目标策略诱导的分布之间存在分布不匹配。几乎所有现有算法都依赖于目标策略诱导分布相对于数据分布的绝对连续性假设，以便通过测度变化利用批量数据校准目标策略。然而，绝对连续性假设在实践中可能被违反，尤其是在状态-动作空间较大或连续的情况下。本文提出了一种新的批量强化学习算法，该算法在连续状态和动作的无限时域马尔可夫决策过程中无需绝对连续性假设。我们将该算法命名为STEEL：奇异性感知强化学习。该算法的动机源于对离线策略评估的新的误差分析，其中我们利用最大均值差异结合分布鲁棒优化，来刻画由可能存在的奇异性导致的离线策略评估误差，并发挥模型外推能力。通过引入悲观思想并在温和条件下，我们为所提算法提供了无需绝对连续性假设的有限样本遗憾保证。与现有算法相比，STEEL仅需最弱的数据覆盖假设，从而大幅提升了批量强化学习的适用性和鲁棒性。大量仿真实验和一项关于个性化定价的真实实验表明，当批量强化学习面临可能的奇异性时，所提方法具有优越性能。