Multistep returns, such as $n$-step returns and $λ$-returns, are commonly used to improve the sample efficiency of reinforcement learning (RL) methods. The variance of the multistep returns becomes the limiting factor in their length; looking too far into the future increases variance and reverses the benefits of multistep learning. In our work, we demonstrate the ability of compound returns -- weighted averages of $n$-step returns -- to reduce variance. We prove for the first time that any compound return with the same contraction modulus as a given $n$-step return has strictly lower variance. We additionally prove that this variance-reduction property improves the finite-sample complexity of temporal-difference learning under linear function approximation. Because general compound returns can be expensive to implement, we introduce two-bootstrap returns which reduce variance while remaining efficient, even when using minibatched experience replay. We conduct experiments showing that compound returns often increase the sample efficiency of $n$-step deep RL agents like DQN and PPO.

翻译：多步回报（如n步回报和λ回报）通常用于提升强化学习方法的样本效率。多步回报的方差成为其长度的限制因素：展望过远的未来会增加方差，从而抵消多步学习的优势。本研究证明了复合回报——即n步回报的加权平均——具有降低方差的能力。我们首次证明，任何与给定n步回报具有相同收缩模数的复合回报，其方差严格更低。此外，我们证明了这种方差降低特性能够改进线性函数逼近下时序差分学习的有限样本复杂度。由于通用复合回报的实现成本较高，我们提出了双自举回报，该方案在保持高效性的同时降低方差，即使在使用小批量经验回放时亦然。实验结果表明，复合回报通常能提升如DQN和PPO等n步深度强化学习智能体的样本效率。