PPO (Proximal Policy Optimization) algorithm has demonstrated excellent performance in many fields, and it is considered as a simple version of TRPO (Trust Region Policy Optimization) algorithm. However, the ratio clipping operation in PPO may not always effectively enforce the trust region constraints, this can be a potential factor affecting the stability of the algorithm. In this paper, we propose SPO (Simple Policy Optimization) algorithm, which introduces a novel clipping method for KL divergence between the old and current policies. SPO can effectively enforce the trust region constraints in almost all environments, while still maintaining the simplicity of a first-order algorithm. Comparative experiments in Atari 2600 environments show that SPO sometimes provides stronger performance than PPO. Code is available at https://github.com/MyRepositories-hub/Simple-Policy-Optimization.

翻译：PPO（近端策略优化）算法已在众多领域展现出卓越性能，且被视为TRPO（信赖域策略优化）算法的简化版本。然而，PPO中的比率裁剪操作可能无法始终有效实施信赖域约束，这可能是影响算法稳定性的潜在因素。本文提出了SPO（简单策略优化）算法，该算法引入了一种新的裁剪方法，用于约束旧策略与新策略之间的KL散度。SPO能够在几乎所有环境中有效实施信赖域约束，同时保持一阶算法的简洁性。在Atari 2600环境中的对比实验表明，SPO有时能提供比PPO更强的性能。代码已开源：https://github.com/MyRepositories-hub/Simple-Policy-Optimization。