We present the first high-probability regret bound for classical online Q-learning in infinite-horizon discounted Markov decision processes, without relying on optimism or bonus terms. We first analyze Boltzmann Q-learning with decaying temperature and show that its regret depends critically on the suboptimality gap of the MDP: for sufficiently large gaps, the regret is sublinear, while for small gaps it deteriorates and can approach linear growth. To address this limitation, we study a Smoothed $ε_n$-Greedy exploration scheme that combines $ε_n$-greedy and Boltzmann exploration, for which we prove a gap-robust regret bound of near-$\tilde{O}(N^{9/10})$. To analyze these algorithms, we develop a high-probability concentration bound for contractive Markovian stochastic approximation with iterate- and time-dependent transition dynamics. This bound may be of independent interest as the contraction factor in our bound is governed by the mixing time and is allowed to converge to one asymptotically.

翻译：我们首次给出了经典在线Q学习在无限时域折扣马尔可夫决策过程中的高概率遗憾界，且无需依赖乐观估计或奖励附加项。我们首先分析了温度衰减的Boltzmann Q学习，证明其遗憾关键取决于MDP的次优间隙：对于充分大的间隙，遗憾呈次线性增长；而对于小间隙，遗憾性能恶化并可能趋近线性增长。为克服此局限，我们研究了一种结合ε_n-贪婪探索与Boltzmann探索的平滑ε_n-贪婪探索策略，并证明了其具有接近$\tilde{O}(N^{9/10})$的间隙鲁棒性遗憾界。为分析这些算法，我们建立了具有迭代依赖性和时变性转移动态的压缩马尔可夫随机逼近过程的高概率集中界。该界中的压缩因子由混合时间控制且允许渐近收敛至一，这一结论可能具有独立的研究价值。