In this paper, we study the finite-time behavior of the TD(0) temporal-difference method with linear function approximation (LFA). We consider on-policy independent and identically distributed (i.i.d.) samples, a constant learning step, and the Polyak-Juditsky averaging method. We establish a new convergence rate, for the Mean-Square Error (MSE) on the approximated function, that is (i) fast in the sense that it admits an optimal dependency in the number of iterations k (i.e., of order 1/k), (ii) robust to ill-conditioning: it only depends on an initial error and modelindependent constants and (iii) sharp up to a multiplicative constant lower than 11. In particular, it does not depend on the smallest eigenvalue of the uncentered covariance matrix of the linear parametrization, unlike all pre-existing O(1/k) rates in the TD(0) literature. We also introduce PCTD(0), a variant of TD(0), which benefits from better convergence properties under an additional assumption of strong mixing on the Markov Chain.

翻译：本文研究了基于线性函数逼近（LFA）的TD(0)时序差分方法的有限时间行为。我们考虑策略内独立同分布（i.i.d.）样本、恒定学习步长以及Polyak-Juditsky平均方法。针对近似函数的均方误差（MSE），我们建立了新的收敛率，该收敛率满足：（i）快速性，即对迭代次数k具有最优依赖性（即1/k阶）；（ii）对病态条件的鲁棒性：仅依赖于初始误差和模型无关常数；（iii）精确性，其乘法常数严格小于11。特别地，与TD(0)文献中所有现有的O(1/k)收敛率不同，该收敛率不依赖于线性参数化非中心协方差矩阵的最小特征值。我们还提出了PCTD(0)算法——TD(0)的一种变体，该变体在马尔可夫链满足强混合性的额外假设下具有更优的收敛性质。