We investigate the finite-time convergence properties of Temporal Difference (TD) learning with linear function approximation, a cornerstone of reinforcement learning. We are interested in the so-called ``robust'' setting, where the convergence guarantee does not depend on the potential function's minimal curvature. While prior work has established convergence guarantees in this setting, these results typically rely on the artificial assumption that each iterate is projected onto a bounded set. Removing such a condition was left as an open problem by Bhandari et al. (COLT'18), hypothesizing the need for additional ``regularity conditions''. In this paper, we show that the simple unprojected TD(0) converges with a rate of $\widetilde{\mathcal{O}}\left(\frac{\|θ^*\|^2_2}{\sqrt{T}}\right)$ in expectation, even in the presence of Markovian noise. We do not require an additional regularity condition, but only a minor polylog correction to the learning rate. Our analysis reveals a novel self-bounding property of the TD updates and exploits it to guarantee bounded iterates.

翻译：我们研究了强化学习基石——带线性函数逼近的时序差分学习（Temporal Difference, TD）的有限时间收敛性质。本文关注所谓的“鲁棒”场景，即收敛性保证不依赖于势函数的最小曲率。尽管已有工作在该场景下建立了收敛性保证，但这些结果通常依赖于将每次迭代投影到有界集上的人为假设。Bhandari 等人（COLT'18）将移除该条件作为开放问题提出，并假设需要额外的“正则条件”。本文证明，即使在马尔可夫噪声存在的情况下，简单的无投影 TD(0) 算法具有 $\widetilde{\mathcal{O}}\left(\frac{\|θ^*\|^2_2}{\sqrt{T}}\right)$ 的期望收敛速率。我们无需额外正则条件，仅需对学习率进行微小的 polylog 修正。我们的分析揭示了 TD 更新的一种新颖的自有界性质，并利用该性质确保了迭代变量的有界性。