Policy-based reinforcement learning algorithms are widely used in various fields. Among them, mainstream policy optimization algorithms such as TRPO and PPO introduce importance sampling into policy iteration, which allows the reuse of historical data. However, this can also lead to a high variance of the surrogate objective and indirectly affects the stability and convergence of the algorithm. In this paper, we first derived an upper bound of the surrogate objective variance, which can grow quadratically with the increase of the surrogate objective. Next, we proposed the dropout technique to avoid the excessive increase of the surrogate objective variance caused by importance sampling. Then, we introduced a general reinforcement learning framework applicable to mainstream policy optimization methods, and applied the dropout technique to the PPO algorithm to obtain the D-PPO variant. Finally, we conduct comparative experiments between D-PPO and PPO algorithms in the Atari 2600 environment, and the results show that D-PPO achieved significant performance improvements compared to PPO, and effectively limited the excessive increase of the surrogate objective variance during training.

翻译：基于策略的强化学习算法广泛应用于各个领域。其中，TRPO和PPO等主流策略优化算法将重要性采样引入策略迭代，从而允许重复使用历史数据。然而，这也会导致替代目标的高方差，间接影响算法的稳定性和收敛性。本文首先推导了替代目标方差的上界，该上界可能随替代目标的增加呈二次方增长。接着，我们提出了dropout技术，以避免重要性采样导致替代目标方差过度增大。然后，我们引入了一个适用于主流策略优化方法的通用强化学习框架，并将dropout技术应用于PPO算法，得到了D-PPO变体。最后，我们在Atari 2600环境中对D-PPO和PPO算法进行了对比实验，结果表明D-PPO相比PPO取得了显著的性能提升，并有效限制了训练过程中替代目标方差的过度增大。