Q-learning played a foundational role in the field reinforcement learning (RL). However, TD algorithms with off-policy data, such as Q-learning, or nonlinear function approximation like deep neural networks require several additional tricks to stabilise training, primarily a large replay buffer and target networks. Unfortunately, the delayed updating of frozen network parameters in the target network harms the sample efficiency and, similarly, the large replay buffer introduces memory and implementation overheads. In this paper, we investigate whether it is possible to accelerate and simplify off-policy TD training while maintaining its stability. Our key theoretical result demonstrates for the first time that regularisation techniques such as LayerNorm can yield provably convergent TD algorithms without the need for a target network or replay buffer, even with off-policy data. Empirically, we find that online, parallelised sampling enabled by vectorised environments stabilises training without the need for a large replay buffer. Motivated by these findings, we propose PQN, our simplified deep online Q-Learning algorithm. Surprisingly, this simple algorithm is competitive with more complex methods like: Rainbow in Atari, PPO-RNN in Craftax, QMix in Smax, and can be up to 50x faster than traditional DQN without sacrificing sample efficiency. In an era where PPO has become the go-to RL algorithm, PQN reestablishes off-policy Q-learning as a viable alternative.

翻译：Q学习在强化学习（RL）领域发挥了奠基性作用。然而，使用离策略数据（如Q学习）或非线性函数近似（如深度神经网络）的时序差分（TD）算法需要多种额外技巧来稳定训练，主要包括大型回放缓冲区和目标网络。遗憾的是，目标网络中冻结网络参数的延迟更新会损害样本效率；同样，大型回放缓冲区会带来内存与实现开销。本文探究在保持稳定性的前提下，能否加速并简化离策略TD训练。我们的关键理论成果首次证明：即使使用离策略数据，LayerNorm等正则化技术也能产生可证明收敛的TD算法，且无需目标网络或回放缓冲区。实证研究表明，通过向量化环境实现的在线并行采样能够稳定训练，而无需大型回放缓冲区。基于这些发现，我们提出简化深度在线Q学习算法PQN。令人惊讶的是，这种简单算法在Atari中与Rainbow、Craftax中与PPO-RNN、Smax中与QMix等复杂方法性能相当，且相比传统DQN速度提升可达50倍而不损失样本效率。在PPO已成为主流RL算法的时代，PQN重新确立了离策略Q学习作为可行替代方案的地位。