$Q$-learning is one of the most fundamental reinforcement learning (RL) algorithms. Despite its widespread success in various applications, it is prone to overestimation bias in the $Q$-learning update. To address this issue, double $Q$-learning employs two independent $Q$-estimators which are randomly selected and updated during the learning process. This paper proposes a modified double $Q$-learning, called simultaneous double $Q$-learning (SDQ), with its finite-time analysis. SDQ eliminates the need for random selection between the two $Q$-estimators, and this modification allows us to analyze double $Q$-learning through the lens of a novel switching system framework facilitating efficient finite-time analysis. Empirical studies demonstrate that SDQ converges faster than double $Q$-learning while retaining the ability to mitigate the maximization bias. Finally, we derive a finite-time expected error bound for SDQ.

翻译：Q学习是最基础的强化学习（RL）算法之一。尽管在各种应用中取得了广泛成功，但Q学习在更新过程中容易产生高估偏差。为解决这一问题，双Q学习采用两个独立的Q估计器，并在学习过程中随机选择和更新。本文提出一种改进的双Q学习方法，称为同步双Q学习（SDQ），并给出其有限时间分析。SDQ消除了在两个Q估计器之间随机选择的需求，这一改进使我们能够通过一种新颖的切换系统框架来分析双Q学习，从而促进高效的有限时间分析。实证研究表明，SDQ比双Q学习收敛更快，同时保持缓解最大化偏差的能力。最后，我们推导了SDQ的有限时间期望误差界。