We study the Non-Stationary Reinforcement Learning (RL) under distribution shifts in both finite-horizon episodic and infinite-horizon discounted Markov Decision Processes (MDPs). In the finite-horizon case, the transition functions may suddenly change at a particular episode. In the infinite-horizon setting, such changes can occur at an arbitrary time step during the agent's interaction with the environment. While the Q-learning Upper Confidence Bound algorithm (QUCB) can discover a proper policy during learning, due to the distribution shifts, this policy can exploit sub-optimal rewards after the shift happens. To address this issue, we propose Density-QUCB (DQUCB), a shift-aware Q-learning~UCB algorithm, which uses a transition density function to detect distribution shifts, then leverages its likelihood to enhance the uncertainty estimation quality of Q-learning~UCB, resulting in a balance between exploration and exploitation. Theoretically, we prove that our oracle DQUCB achieves a better regret guarantee than QUCB. Empirically, our DQUCB enjoys the computational efficiency of model-free RL and outperforms QUCB baselines by having a lower regret across RL tasks, as well as a real-world COVID-19 patient hospital allocation task using a Deep-Q-learning architecture.

翻译：本研究探讨了在有限时域片段和无限时域折扣马尔可夫决策过程（MDPs）中，存在分布偏移的非平稳强化学习（RL）问题。在有限时域场景下，转移函数可能在特定片段处发生突变；而在无限时域设定中，此类变化可在智能体与环境交互的任意时间步发生。尽管Q学习上置信界算法（QUCB）能在学习过程中发现合适的策略，但由于分布偏移的存在，该策略在偏移发生后可能仅能利用次优奖励。为解决此问题，我们提出了Density-QUCB（DQUCB）——一种具有偏移感知能力的Q学习~UCB算法。该算法通过转移密度函数检测分布偏移，并利用其似然性提升Q学习~UCB的不确定性估计质量，从而实现探索与利用的平衡。理论上，我们证明了所提出的预言机DQUCB能获得比QUCB更优的遗憾界保证。实证研究表明，DQUCB兼具模型无关RL的计算效率优势，并在多个RL任务中通过更低的遗憾值优于QUCB基线方法；同时，在使用深度Q学习架构的真实世界COVID-19患者医院分配任务中也表现出更优性能。