Holdout validation and hyperparameter tuning from data is a long-standing problem in offline reinforcement learning (RL). A standard framework is to use off-policy evaluation (OPE) methods to evaluate and select the policies, but OPE either incurs exponential variance (e.g., importance sampling) or has hyperparameters on their own (e.g., FQE and model-based). We focus on hyperparameter tuning for OPE itself, which is even more under-investigated. Concretely, we select among candidate value functions ("model-free") or dynamics ("model-based") to best assess the performance of a target policy. Concretely, we select among candidate value functions (``model-free'') or dynamics models (``model-based'') to best assess the performance of a target policy. We develop: (1) new model-free and model-based selectors with theoretical guarantees, and (2) a new experimental protocol for empirically evaluating them. Compared to the model-free protocol in prior works, our new protocol allows for more stable generation and better control of candidate value functions in an optimization-free manner, and evaluation of model-free and model-based methods alike. We exemplify the protocol on Gym-Hopper, and find that our new model-free selector, LSTD-Tournament, demonstrates promising empirical performance.

翻译：在离线强化学习（RL）中，利用数据进行留出验证与超参数调优是一个长期存在的问题。标准框架是使用离线策略评估（OPE）方法来评估和选择策略，但OPE方法要么会产生指数级方差（例如重要性采样），要么自身就带有超参数（例如FQE和基于模型的方法）。我们聚焦于OPE本身的超参数调优，这一方向的研究更为缺乏。具体而言，我们从候选价值函数（“无模型”方法）或动力学模型（“基于模型”方法）中进行选择，以最佳地评估目标策略的性能。我们开发了：（1）具有理论保证的新型无模型与基于模型选择器；（2）用于对其进行实证评估的新实验协议。与先前工作中的无模型协议相比，我们的新协议能够以无需优化的方式更稳定地生成并更好地控制候选价值函数，并可同时评估无模型与基于模型的方法。我们在Gym-Hopper环境中对该协议进行了示例验证，发现我们提出的新型无模型选择器——LSTD-Tournament——展现出具有前景的实证性能。