In this work, we present the first finite-time analysis of the Q-learning algorithm under time-varying learning policies (i.e., on-policy sampling) with minimal assumptions -- specifically, assuming only the existence of a policy that induces an irreducible Markov chain over the state space. We establish a last-iterate convergence rate for $\mathbb{E}[\|Q_k - Q^*\|_\infty^2]$, implying a sample complexity of order $O(1/\epsilon^2)$ for achieving $\mathbb{E}[\|Q_k - Q^*\|_\infty] \le \epsilon$, matching that of off-policy Q-learning but with a worse dependence on exploration-related parameters. We also derive an explicit rate for $\mathbb{E}[\|Q^{\pi_k} - Q^*\|_\infty^2]$, where $\pi_k$ is the learning policy at iteration $k$. These results reveal that on-policy Q-learning exhibits weaker exploration than its off-policy counterpart but enjoys an exploitation advantage, as its policy converges to an optimal one rather than remaining fixed. Numerical simulations corroborate our theory. Technically, the combination of time-varying learning policies (which induce rapidly time-inhomogeneous Markovian noise) and the minimal assumption on exploration presents significant analytical challenges. To address these challenges, we employ a refined approach that leverages the Poisson equation to decompose the Markovian noise corresponding to the lazy transition matrix into a martingale-difference term and residual terms. To control the residual terms under time inhomogeneity, we perform a sensitivity analysis of the Poisson equation solution with respect to both the Q-function estimate and the learning policy. These tools may further facilitate the analysis of general reinforcement learning algorithms with rapidly time-varying learning policies -- such as single-timescale actor--critic methods and learning-in-games algorithms -- and are of independent interest.

翻译：本文首次在最小假设条件下——具体而言，仅假设存在一个能在状态空间上诱导不可约马尔可夫链的策略——对时变学习策略（即同策略采样）下的Q学习算法进行了有限时间分析。我们建立了$\mathbb{E}[\|Q_k - Q^*\|_\infty^2]$的最终迭代收敛速率，这意味着达到$\mathbb{E}[\|Q_k - Q^*\|_\infty] \le \epsilon$所需的样本复杂度为$O(1/\epsilon^2)$量级，与异策略Q学习的样本复杂度相匹配，但在探索相关参数上的依赖性更差。我们还推导了$\mathbb{E}[\|Q^{\pi_k} - Q^*\|_\infty^2]$的显式速率，其中$\pi_k$是第$k$次迭代时的学习策略。这些结果表明，同策略Q学习相比其异策略对应方法展现出较弱的探索性，但享有利用优势，因为其策略会收敛至最优策略而非保持固定。数值模拟验证了我们的理论。从技术上讲，时变学习策略（其诱导快速时非齐次的马尔可夫噪声）与关于探索的最小假设相结合，带来了显著的分析挑战。为应对这些挑战，我们采用了一种精细方法，利用泊松方程将与惰性转移矩阵对应的马尔可夫噪声分解为一个鞅差项和若干残差项。为了在时非齐次性下控制残差项，我们对泊松方程解关于Q函数估计和学习策略进行了灵敏度分析。这些工具可能进一步促进对具有快速时变学习策略的通用强化学习算法——例如单时间尺度行动者-评论家方法和博弈学习算法——的分析，并具有独立的研究价值。