The two-time scale nature of SAC, which is an actor-critic algorithm, is characterised by the fact that the critic estimate has not converged for the actor at any given time, but since the critic learns faster than the actor, it ensures eventual consistency between the two. Various strategies have been introduced in literature to learn better gradient estimates to help achieve better convergence. Since gradient estimates depend upon the critic, we posit that improving the critic can provide a better gradient estimate for the actor at each time. Utilizing this, we propose Soft Actor Retrospective Critic (SARC), where we augment the SAC critic loss with another loss term - retrospective loss - leading to faster critic convergence and consequently, better policy gradient estimates for the actor. An existing implementation of SAC can be easily adapted to SARC with minimal modifications. Through extensive experimentation and analysis, we show that SARC provides consistent improvement over SAC on benchmark environments. We plan to open-source the code and all experiment data at: https://github.com/sukritiverma1996/SARC.

翻译：SAC作为一种演员-评论家算法，其双时间尺度特性表现为：在任意时刻评论家估计值尚未对演员收敛，但由于评论家学习速度快于演员，最终确保两者一致性。现有文献已提出多种策略来学习更优的梯度估计以实现更好的收敛性。鉴于梯度估计依赖于评论家，我们提出改进评论家可为演员提供更优的逐时梯度估计。基于此，我们提出了软演员回溯评论家（SARC），通过向SAC评论家损失函数中增加另一损失项——回溯损失——来加速评论家收敛，进而为演员提供更优的策略梯度估计。SAC的现有实现可经最小修改轻松适配至SARC。通过大量实验与分析，我们证明SARC在基准测试环境中相比SAC具有持续改进效果。我们计划将代码及所有实验数据开源在：https://github.com/sukritiverma1996/SARC。