Fitted Q-evaluation (FQE) is a foundational method for off-policy evaluation in reinforcement learning, but existing theory typically relies on Bellman completeness of the function class, a condition often violated in practice. This reliance is due to a fundamental norm mismatch: the Bellman operator is gamma-contractive in the L^2 norm induced by the target policy's stationary distribution, whereas standard FQE fits Bellman regressions under the behavior distribution. To resolve this mismatch, we reweight each Bellman regression step by an estimate of the stationary density ratio, inspired by emphatic weighting in temporal-difference learning. This makes the update behave as if it were performed under the target stationary distribution, restoring contraction without Bellman completeness while preserving the simplicity of regression-based evaluation. Illustrative experiments, including Baird's classical counterexample, show that stationary weighting can stabilize FQE under off-policy sampling.

翻译：拟合Q评估（FQE）是强化学习中离线策略评估的基础方法，但现有理论通常依赖于函数类的贝尔曼完备性，而该条件在实践中常被违反。这种依赖性源于基本的范数失配：贝尔曼算子在目标策略平稳分布诱导的L²范数下具有γ-收缩性，而标准FQE在行为分布下进行贝尔曼回归拟合。为解决这一失配问题，我们借鉴时序差分学习中的强调加权方法，通过对每个贝尔曼回归步骤施加平稳密度比的估计值进行重加权。这使得更新过程仿佛在目标平稳分布下进行，从而在保持基于回归的评估简洁性的同时，无需贝尔曼完备性即可恢复收缩特性。包含Baird经典反例在内的 illustrative 实验表明，平稳加权能有效稳定离线策略采样下的FQE方法。