We analyse quantile temporal-difference learning (QTD), a distributional reinforcement learning algorithm that has proven to be a key component in several successful large-scale applications of reinforcement learning. Despite these empirical successes, a theoretical understanding of QTD has proven elusive until now. Unlike classical TD learning, which can be analysed with standard stochastic approximation tools, QTD updates do not approximate contraction mappings, are highly non-linear, and may have multiple fixed points. The core result of this paper is a proof of convergence to the fixed points of a related family of dynamic programming procedures with probability 1, putting QTD on firm theoretical footing. The proof establishes connections between QTD and non-linear differential inclusions through stochastic approximation theory and non-smooth analysis.

翻译：本文分析了分位数时序差分学习（QTD），这是一种分布式强化学习算法，已被证明是多个成功的大规模强化学习应用中的关键组成部分。尽管取得了这些经验上的成功，但对QTD的理论理解至今仍难以实现。与可通过标准随机逼近工具分析的经典TD学习不同，QTD更新不近似压缩映射，具有高度非线性特性，且可能存在多个不动点。本文的核心结果是以概率1证明了QTD收敛至相关动态规划程序族的不动点，从而为QTD奠定了坚实的理论基础。该证明通过随机逼近理论和非光滑分析，建立了QTD与非线性微分包含之间的关联。