On-policy algorithms are supposed to be stable, however, sample-intensive yet. Off-policy algorithms utilizing past experiences are deemed to be sample-efficient, nevertheless, unstable in general. Can we design an algorithm that can employ the off-policy data, while exploit the stable learning by sailing along the course of the on-policy walkway? In this paper, we present an actor-critic learning framework that borrows the distributional perspective of interest to evaluate, and cross-breeds two sources of the data for policy improvement, which enables fast learning and can be applied to a wide class of algorithms. In its backbone, the variance reduction mechanisms, such as unified advantage estimator (UAE), that extends generalized advantage estimator (GAE) to be applicable on any state-dependent baseline, and a learned baseline, that is competent to stabilize the policy gradient, are firstly put forward to not merely be a bridge to the action-value function but also distill the advantageous learning signal. Lastly, it is empirically shown that our method improves sample efficiency and interpolates different levels well. Being of an organic whole, its mixture places more inspiration to the algorithm design.

翻译：基于同策略的算法本应具有稳定性，但存在样本效率低下的问题。而利用过往经验的异策略算法虽被认为更省样本，但通常不够稳定。能否设计一种既能利用异策略数据，又能沿同策略路径航行的稳定学习算法？本文提出了一种演员-评论家学习框架，该框架借鉴了用于价值评估的分布视角，并通过交叉融合两种数据源进行策略改进，从而实现了快速学习，可广泛应用于多种算法。其核心架构首次提出了方差缩减机制：扩展广义优势估计器（GAE）使其适用于任意状态相关基线的统一优势估计器（UAE），以及能够稳定策略梯度的可学习基线函数——这些机制不仅充当了动作价值函数的桥梁，更蒸馏出有利的学习信号。实验结果表明，该方法提升了样本效率，并能良好地适配不同等级的任务。作为一个有机整体，其混合机制为算法设计注入了更多灵感。