Actor-critic algorithms address the dual goals of reinforcement learning (RL), policy evaluation and improvement, via two separate function approximators. The practicality of this approach comes at the expense of training instability, caused mainly by the destructive effect of the approximation errors of the critic on the actor. We tackle this bottleneck by employing an existing Probably Approximately Correct (PAC) Bayesian bound for the first time as the critic training objective of the Soft Actor-Critic (SAC) algorithm. We further demonstrate that online learning performance improves significantly when a stochastic actor explores multiple futures by critic-guided random search. We observe our resulting algorithm to compare favorably to the state of the art on multiple classical control and locomotion tasks in terms of both sample efficiency and regret minimization.

翻译：演员-评论家算法通过两个独立的函数近似器来实现强化学习的双重目标：策略评估与改进。该方法的实用性以训练不稳定为代价，这主要由评论家的近似误差对演员造成的破坏性影响所导致。我们首次将现有的可能近似正确（PAC）贝叶斯界作为软演员-评论家（SAC）算法的评论家训练目标，以解决这一瓶颈问题。我们进一步证明，当随机演员通过评论家引导的随机搜索探索多种未来情境时，在线学习性能显著提升。实验表明，在多个经典控制与运动任务中，我们提出的算法在样本效率和遗憾最小化方面均优于现有最优水平。