The current state-of-the-art theoretical analysis of Actor-Critic (AC) algorithms significantly lags in addressing the practical aspects of AC implementations. This crucial gap needs bridging to bring the analysis in line with practical implementations of AC. To address this, we advocate for considering the MMCLG criteria: \textbf{M}ulti-layer neural network parametrization for actor/critic, \textbf{M}arkovian sampling, \textbf{C}ontinuous state-action spaces, the performance of the \textbf{L}ast iterate, and \textbf{G}lobal optimality. These aspects are practically significant and have been largely overlooked in existing theoretical analyses of AC algorithms. In this work, we address these gaps by providing the first comprehensive theoretical analysis of AC algorithms that encompasses all five crucial practical aspects (covers MMCLG criteria). We establish global convergence sample complexity bounds of $\tilde{\mathcal{O}}\left({\epsilon^{-3}}\right)$. We achieve this result through our novel use of the weak gradient domination property of MDP's and our unique analysis of the error in critic estimation.

翻译：当前关于演员-评论家（Actor-Critic, AC）算法的理论分析在解决实际实现问题方面显著滞后。这一关键差距亟待弥合，以使理论分析与AC的实际实现保持一致。为此，我们主张考虑MMCLG标准：演员/评论家采用\textbf{M}ulti-layer神经网络参数化、\textbf{M}arkov采样、\textbf{C}ontinuous状态-动作空间、\textbf{L}ast迭代性能以及\textbf{G}lobal最优性。这些实际方面至关重要，但在现有AC算法理论分析中大多被忽视。在本工作中，我们通过首次提供涵盖所有五个关键实际方面（即满足MMCLG标准）的AC算法全面理论分析来填补这些空白。我们建立了全局收敛样本复杂度界为$\tilde{\mathcal{O}}\left({\epsilon^{-3}}\right)$。这一结果得益于我们对马尔可夫决策过程弱梯度支配性质的新颖应用，以及我们对评论家估计误差的独特分析。