This paper investigates the off-policy evaluation (OPE) problem from a distributional perspective. Rather than focusing solely on the expectation of the total return, as in most existing OPE methods, we aim to estimate the entire return distribution. To this end, we introduce a quantile-based approach for OPE using deep quantile process regression, presenting a novel algorithm called Deep Quantile Process regression-based Off-Policy Evaluation (DQPOPE). We provide new theoretical insights into the deep quantile process regression technique, extending existing approaches that estimate discrete quantiles to estimate a continuous quantile function. A key contribution of our work is the rigorous sample complexity analysis for distributional OPE with deep neural networks, bridging theoretical analysis with practical algorithmic implementations. We show that DQPOPE achieves statistical advantages by estimating the full return distribution using the same sample size required to estimate a single policy value using conventional methods. Empirical studies further show that DQPOPE provides significantly more precise and robust policy value estimates than standard methods, thereby enhancing the practical applicability and effectiveness of distributional reinforcement learning approaches.

翻译：本文从分布视角研究离策略评估问题。与现有大多数离策略评估方法仅关注总回报的期望不同，我们旨在估计完整的回报分布。为此，我们引入了一种基于分位数的离策略评估方法，采用深度分位数过程回归技术，提出了一种名为基于深度分位数过程回归的离策略评估（DQPOPE）的新算法。我们为深度分位数过程回归技术提供了新的理论洞见，将现有的离散分位数估计方法扩展为连续分位数函数的估计。本研究的关键贡献在于，针对深度神经网络的分布离策略评估，提供了严格的样本复杂度分析，架起了理论分析与实际算法实现之间的桥梁。我们证明，DQPOPE 能够在使用与传统方法估计单一策略值所需相同样本量的条件下，通过估计完整回报分布实现统计优势。实证研究进一步表明，DQPOPE 相比标准方法能提供显著更精确、更稳健的策略值估计，从而增强了分布强化学习方法的实际适用性和有效性。