The problem of sample complexity of online reinforcement learning is often studied in the literature without taking into account any partial knowledge about the system dynamics that could potentially accelerate the learning process. In this paper, we study the sample complexity of online Q-learning methods when some prior knowledge about the dynamics is available or can be learned efficiently. We focus on systems that evolve according to an additive disturbance model of the form $S_{h+1} = f(S_h, A_h) + W_h$, where $f$ represents the underlying system dynamics, and $W_h$ are unknown disturbances independent of states and actions. In the setting of finite episodic Markov decision processes with $S$ states, $A$ actions, and episode length $H$, we present an optimistic Q-learning algorithm that achieves $\tilde{\mathcal{O}}(\text{Poly}(H)\sqrt{T})$ regret under perfect knowledge of $f$, where $T$ is the total number of interactions with the system. This is in contrast to the typical $\tilde{\mathcal{O}}(\text{Poly}(H)\sqrt{SAT})$ regret for existing Q-learning methods. Further, if only a noisy estimate $\hat{f}$ of $f$ is available, our method can learn an approximately optimal policy in a number of samples that is independent of the cardinalities of state and action spaces. The sub-optimality gap depends on the approximation error $\hat{f}-f$, as well as the Lipschitz constant of the corresponding optimal value function. Our approach does not require modeling of the transition probabilities and enjoys the same memory complexity as model-free methods.

翻译：在线强化学习的样本复杂度问题在现有文献中常被研究，但通常未考虑可能加速学习过程的任何关于系统动力学的先验知识。本文研究了当存在关于动力学的先验知识或可高效学习此类知识时，在线Q学习方法的样本复杂度。我们关注按加性扰动模型演化的系统，其形式为$S_{h+1} = f(S_h, A_h) + W_h$，其中$f$表示基础系统动力学，$W_h$是与状态和动作独立的未知扰动。在具有$S$个状态、$A$个动作和回合长度$H$的有限回合马尔可夫决策过程设定下，我们提出了一种乐观Q学习算法，该算法在完全知晓$f$的情况下可实现$\tilde{\mathcal{O}}(\text{Poly}(H)\sqrt{T})$的遗憾值，其中$T$是与系统的总交互次数。这与现有Q学习方法典型的$\tilde{\mathcal{O}}(\text{Poly}(H)\sqrt{SAT})$遗憾值形成对比。此外，若仅可获得$f$的噪声估计$\hat{f}$，我们的方法能够以独立于状态和动作空间基数的样本量学习近似最优策略。次优性差距取决于近似误差$\hat{f}-f$以及相应最优值函数的Lipschitz常数。我们的方法无需对转移概率进行建模，且保持与无模型方法相同的内存复杂度。