We analyze the convergence rate of the unregularized natural policy gradient algorithm with log-linear policy parametrizations in infinite-horizon discounted Markov decision processes. In the deterministic case, when the Q-value is known and can be approximated by a linear combination of a known feature function up to a bias error, we show that a geometrically-increasing step size yields a linear convergence rate towards an optimal policy. We then consider the sample-based case, when the best representation of the Q- value function among linear combinations of a known feature function is known up to an estimation error. In this setting, we show that the algorithm enjoys the same linear guarantees as in the deterministic case up to an error term that depends on the estimation error, the bias error, and the condition number of the feature covariance matrix. Our results build upon the general framework of policy mirror descent and extend previous findings for the softmax tabular parametrization to the log-linear policy class.

翻译：我们分析了在无限时域折扣马尔可夫决策过程中，采用对数线性策略参数化的未正则化自然策略梯度算法的收敛速率。在确定性情况下，当Q值已知且可通过已知特征函数的线性组合逼近至某一偏差误差时，我们证明几何递增步长能实现向最优策略的线性收敛。随后考虑基于样本的情形：当已知特征函数线性组合对Q值函数的最佳表示已知但存在估计误差时，我们证明该算法与确定性情况享有相同的线性保证，仅额外依赖于估计误差、偏差误差以及特征协方差矩阵条件数的误差项。我们的结果基于策略镜像下降的通用框架，并将先前关于softmax表格型参数化的结论扩展至对数线性策略类。