The problem of sample complexity of online reinforcement learning is often studied in the literature without taking into account any partial knowledge about the system dynamics that could potentially accelerate the learning process. In this paper, we study the sample complexity of online Q-learning methods when some prior knowledge about the dynamics is available or can be learned efficiently. We focus on systems that evolve according to an additive disturbance model of the form $S_{h+1} = f(S_h, A_h) + W_h$, where $f$ represents the underlying system dynamics, and $W_h$ are unknown disturbances independent of states and actions. In the setting of finite episodic Markov decision processes with $S$ states, $A$ actions, and episode length $H$, we present an optimistic Q-learning algorithm that achieves $\tilde{\mathcal{O}}(\text{Poly}(H)\sqrt{T})$ regret under perfect knowledge of $f$, where $T$ is the total number of interactions with the system. This is in contrast to the typical $\tilde{\mathcal{O}}(\text{Poly}(H)\sqrt{SAT})$ regret for existing Q-learning methods. Further, if only a noisy estimate $\hat{f}$ of $f$ is available, our method can learn an approximately optimal policy in a number of samples that is independent of the cardinalities of state and action spaces. The sub-optimality gap depends on the approximation error $\hat{f}-f$, as well as the Lipschitz constant of the corresponding optimal value function. Our approach does not require modeling of the transition probabilities and enjoys the same memory complexity as model-free methods.

翻译：在线强化学习的样本复杂度问题通常在研究文献中未考虑可能加速学习过程的系统动力学部分知识。本文研究当具备或能高效学习某些动力学先验知识时，在线Q-学习方法的样本复杂度。我们聚焦于遵循加性扰动模型演化的系统，其形式为 $S_{h+1} = f(S_h, A_h) + W_h$，其中 $f$ 表示底层系统动力学，$W_h$ 是与状态和动作无关的未知扰动。在有限回合马尔可夫决策过程（状态空间大小为 $S$，动作空间大小为 $A$，回合长度为 $H$）的设置下，我们提出一种乐观Q-学习算法，在完全已知 $f$ 的条件下，达到 $\tilde{\mathcal{O}}(\text{Poly}(H)\sqrt{T})$ 的遗憾值，其中 $T$ 是与系统的总交互次数。这与现有Q-学习方法典型的 $\tilde{\mathcal{O}}(\text{Poly}(H)\sqrt{SAT})$ 遗憾值形成对比。进一步地，若仅能获得 $f$ 的含噪估计 $\hat{f}$，我们的方法可在样本数量与状态和动作空间基数无关的前提下学习近似最优策略。次优性差距取决于近似误差 $\hat{f}-f$ 以及相应最优值函数的利普希茨常数。该方法无需对转移概率进行建模，且具有与无模型方法相同的存储复杂度。