The problem of sample complexity of online reinforcement learning is often studied in the literature without taking into account any partial knowledge about the system dynamics that could potentially accelerate the learning process. In this paper, we study the sample complexity of online Q-learning methods when some prior knowledge about the dynamics is available or can be learned efficiently. We focus on systems that evolve according to an additive disturbance model of the form $S_{h+1} = f(S_h, A_h) + W_h$, where $f$ represents the underlying system dynamics, and $W_h$ are unknown disturbances independent of states and actions. In the setting of finite episodic Markov decision processes with $S$ states, $A$ actions, and episode length $H$, we present an optimistic Q-learning algorithm that achieves $\tilde{\mathcal{O}}(\text{Poly}(H)\sqrt{T})$ regret under perfect knowledge of $f$, where $T$ is the total number of interactions with the system. This is in contrast to the typical $\tilde{\mathcal{O}}(\text{Poly}(H)\sqrt{SAT})$ regret for existing Q-learning methods. Further, if only a noisy estimate $\hat{f}$ of $f$ is available, our method can learn an approximately optimal policy in a number of samples that is independent of the cardinalities of state and action spaces. The sub-optimality gap depends on the approximation error $\hat{f}-f$, as well as the Lipschitz constant of the corresponding optimal value function. Our approach does not require modeling of the transition probabilities and enjoys the same memory complexity as model-free methods.

翻译：在线强化学习的样本复杂度问题通常在文献中未考虑任何关于系统动力学的部分知识，而这些知识可能加速学习过程。本文研究了当动力学部分先验知识可用或可高效学习时，在线Q学习方法的样本复杂度。我们聚焦于遵循加性扰动模型 $S_{h+1} = f(S_h, A_h) + W_h$ 演化的系统，其中 $f$ 表示底层系统动力学，$W_h$ 是与状态和动作无关的未知扰动。在具有 $S$ 个状态、$A$ 个动作和回合长度 $H$ 的有限回合马尔可夫决策过程设定下，我们提出了一种乐观Q学习算法，在完全知晓 $f$ 的情况下达到 $\tilde{\mathcal{O}}(\text{Poly}(H)\sqrt{T})$ 的遗憾值，其中 $T$ 为与系统交互的总次数。这与现有Q学习方法通常的 $\tilde{\mathcal{O}}(\text{Poly}(H)\sqrt{SAT})$ 遗憾值形成对比。进一步地，若仅获得 $f$ 的含噪声估计 $\hat{f}$，我们的方法可以在样本数量与状态和动作空间基数无关的条件下学习近似最优策略。次优性差距取决于近似误差 $\hat{f}-f$ 以及相应最优值函数的利普希茨常数。我们的方法无需对转移概率进行建模，并具有与无模型方法相同的记忆复杂度。