Solving partially observable Markov decision processes (POMDPs) remains a fundamental challenge in reinforcement learning (RL), primarily due to the curse of dimensionality induced by the non-stationarity of optimal policies. In this work, we study a natural actor-critic (NAC) algorithm that integrates recurrent neural network (RNN) architectures into a natural policy gradient (NPG) method and a temporal difference (TD) learning method. This framework leverages the representational capacity of RNNs to address non-stationarity in RL to solve POMDPs while retaining the statistical and computational efficiency of natural gradient methods in RL. We provide non-asymptotic theoretical guarantees for this method, including bounds on sample and iteration complexity to achieve global optimality up to function approximation. Additionally, we characterize pathological cases that stem from long-term dependencies, thereby explaining limitations of RNN-based policy optimization for POMDPs.

翻译：求解部分可观测马尔可夫决策过程（POMDPs）仍然是强化学习（RL）中的一个基础性挑战，这主要源于最优策略的非平稳性所引发的维度灾难。在本研究中，我们探讨了一种自然执行者-评论者（NAC）算法，该算法将循环神经网络（RNN）架构整合到自然策略梯度（NPG）方法与时间差分（TD）学习方法中。该框架利用RNN的表征能力来处理RL中的非平稳性问题，以求解POMDPs，同时保留了自然梯度方法在RL中的统计与计算效率。我们为此方法提供了非渐近的理论保证，包括达到函数逼近意义下的全局最优性所需的样本与迭代复杂度界限。此外，我们刻画了源于长期依赖的病态情况，从而解释了基于RNN的策略优化方法在求解POMDPs时的局限性。