Estimation of value in policy gradient methods is a fundamental problem. Generalized Advantage Estimation (GAE) is an exponentially-weighted estimator of an advantage function similar to $\lambda$-return. It substantially reduces the variance of policy gradient estimates at the expense of bias. In practical applications, a truncated GAE is used due to the incompleteness of the trajectory, which results in a large bias during estimation. To address this challenge, instead of using the entire truncated GAE, we propose to take a part of it when calculating updates, which significantly reduces the bias resulting from the incomplete trajectory. We perform experiments in MuJoCo and $\mu$RTS to investigate the effect of different partial coefficient and sampling lengths. We show that our partial GAE approach yields better empirical results in both environments.

翻译：在策略梯度方法中，价值估计是一个基本问题。广义优势估计（GAE）是一种与$\lambda$-return类似的指数加权优势函数估计器，它通过引入偏差来显著降低策略梯度估计的方差。在实际应用中，由于轨迹的不完整性，通常使用截断的GAE，这会导致估计过程中产生较大偏差。为解决这一挑战，我们提出在计算更新时，不使用完整的截断GAE，而是取其一部分，从而显著减少由不完整轨迹引起的偏差。我们在MuJoCo和$\mu$RTS中进行实验，研究不同部分系数和采样长度的影响。实验结果表明，我们的部分GAE方法在两种环境中均能获得更好的经验结果。