Theoretical guarantees in reinforcement learning (RL) are known to suffer multiplicative blow-up factors with respect to the misspecification error of function approximation. Yet, the nature of such \emph{approximation factors} -- especially their optimal form in a given learning problem -- is poorly understood. In this paper we study this question in linear off-policy value function estimation, where many open questions remain. We study the approximation factor in a broad spectrum of settings, such as with the weighted $L_2$-norm (where the weighting is the offline state distribution), the $L_\infty$ norm, the presence vs. absence of state aliasing, and full vs. partial coverage of the state space. We establish the optimal asymptotic approximation factors (up to constants) for all of these settings. In particular, our bounds identify two instance-dependent factors for the $L_2(\mu)$ norm and only one for the $L_\infty$ norm, which are shown to dictate the hardness of off-policy evaluation under misspecification.

翻译：在强化学习（RL）中，已知理论保证会因函数近似中的设定错误而承受乘法放大因子。然而，这些“近似因子”的性质——尤其是在给定学习问题中的最优形式——尚未得到充分理解。本文在线性离策略价值函数估计中研究该问题，该领域仍存在许多未解疑问。我们在广泛的情境下研究近似因子，例如加权 $L_2$ 范数（其中权重为离线状态分布）、$L_\infty$ 范数、状态混叠的存在与否、以及状态空间的完全覆盖与部分覆盖。我们为所有这些情境建立了最优渐近近似因子（至常数项）。特别地，我们的界识别出 $L_2(\mu)$ 范数下的两个实例依赖因子，而 $L_\infty$ 范数下仅有一个因子，这些因子决定了在设定错误下离策略评估的难度。