Proximal Policy Optimization algorithm employing a clipped surrogate objective (PPO-Clip) is a prominent exemplar of the policy optimization methods. However, despite its remarkable empirical success, PPO-Clip lacks theoretical substantiation to date. In this paper, we contribute to the field by establishing the first global convergence results of a PPO-Clip variant in both tabular and neural function approximation settings. Our findings highlight the $O(1/\sqrt{T})$ min-iterate convergence rate specifically in the context of neural function approximation. We tackle the inherent challenges in analyzing PPO-Clip through three central concepts: (i) We introduce a generalized version of the PPO-Clip objective, illuminated by its connection with the hinge loss. (ii) Employing entropic mirror descent, we establish asymptotic convergence for tabular PPO-Clip with direct policy parameterization. (iii) Inspired by the tabular analysis, we streamline convergence analysis by introducing a two-step policy improvement approach. This decouples policy search from complex neural policy parameterization using a regression-based update scheme. Furthermore, we gain deeper insights into the efficacy of PPO-Clip by interpreting these generalized objectives. Our theoretical findings also mark the first characterization of the influence of the clipping mechanism on PPO-Clip convergence. Importantly, the clipping range affects only the pre-constant of the convergence rate.

翻译：采用裁剪替代目标的近端策略优化算法（PPO-Clip）是策略优化方法的典型代表。然而，尽管其在实际应用中取得了显著成功，PPO-Clip至今缺乏理论支撑。本文通过建立PPO-Clip变体在表格设置和神经函数逼近设置下的首个全局收敛结果，为该领域做出贡献。我们的发现揭示了神经函数逼近场景中$O(1/\sqrt{T})$的极小迭代收敛率。我们通过三个核心概念应对分析PPO-Clip的内在挑战：(i) 引入广义PPO-Clip目标，阐明其与铰链损失的联系；(ii) 利用熵镜像下降方法，建立直接策略参数化下表格型PPO-Clip的渐近收敛性；(iii) 受表格分析启发，通过两步策略改进方法简化收敛性分析，该方法使用基于回归的更新机制将策略搜索与复杂神经策略参数化解耦。此外，我们通过解释这些广义目标，更深入地理解PPO-Clip的有效性。我们的理论发现还首次刻画了裁剪机制对PPO-Clip收敛性的影响特征。重要的是，裁剪范围仅影响收敛率的前置常数。