This paper investigates the use of prior computation to estimate the value function to improve sample efficiency in on-policy policy gradient methods in reinforcement learning. Our approach is to estimate the value function from prior computations, such as from the Q-network learned in DQN or the value function trained for different but related environments. In particular, we learn a new value function for the target task while combining it with a value estimate from the prior computation. Finally, the resulting value function is used as a baseline in the policy gradient method. This use of a baseline has the theoretical property of reducing variance in gradient computation and thus improving sample efficiency. The experiments show the successful use of prior value estimates in various settings and improved sample efficiency in several tasks.

翻译：本文研究在强化学习的同策略策略梯度方法中，利用先验计算估计值函数以提高样本效率。我们的方法是从先验计算中估计值函数，例如从DQN中学习的Q网络或为不同但相关环境训练的值函数中获取。具体而言，我们为目标任务学习新的值函数，同时将其与先验计算得到的值估计相结合。最终的值函数被用作策略梯度方法中的基线。这种基线的使用具有降低梯度计算方差的特性，从而提升样本效率。实验表明，在各种设置下成功利用先验值估计，并在多个任务中提高了样本效率。