We study off-policy reinforcement learning for controlling continuous-time Markov diffusion processes with discrete-time observations and actions. We consider model-free algorithms with function approximation that learn value and advantage functions directly from data, without unrealistic structural assumptions on the dynamics. Leveraging the ellipticity of the diffusions, we establish a new class of Hilbert-space positive definiteness and boundedness properties for the Bellman operators. Based on these properties, we propose the Sobolev-prox fitted $q$-learning algorithm, which learns value and advantage functions by iteratively solving least-squares regression problems. We derive oracle inequalities for the estimation error, governed by (i) the best approximation error of the function classes, (ii) their localized complexity, (iii) exponentially decaying optimization error, and (iv) numerical discretization error. These results identify ellipticity as a key structural property that renders reinforcement learning with function approximation for Markov diffusions no harder than supervised learning.

翻译：我们研究了离策略强化学习，用于控制具有离散时间观测和动作的连续时间马尔可夫扩散过程。我们考虑采用函数逼近的无模型算法，直接从数据中学习价值和优势函数，无需对动态过程设定不切实际的结构假设。利用扩散过程的椭圆性，我们建立了贝尔曼算子的一类新的希尔伯特空间正定性和有界性性质。基于这些性质，我们提出了Sobolev近端拟合Q学习算法，通过迭代求解最小二乘回归问题来学习价值和优势函数。我们推导了估计误差的置信区间不等式，该误差由以下因素决定：(i) 函数类的最佳逼近误差；(ii) 其局域化复杂度；(iii) 指数衰减的优化误差；(iv) 数值离散误差。这些结果将椭圆性确定为一种关键结构性质，使得马尔可夫扩散过程的函数逼近强化学习不再比监督学习困难。