To learn approximately optimal acting policies for decision problems, modern Actor Critic algorithms rely on deep Neural Networks (DNNs) to parameterize the acting policy and greedification operators to iteratively improve it. The reliance on DNNs suggests an improvement that is gradient based, which is per step much less greedy than the improvement possible by greedier operators such as the greedy update used by Q-learning algorithms. On the other hand, slow changes to the policy can also be beneficial for the stability of the learning process, resulting in a tradeoff between greedification and stability. To better address this tradeoff, we propose to decouple the acting policy from the policy evaluated by the critic. This allows the agent to separately improve the critic's policy (e.g. value improvement) with greedier updates while maintaining the slow gradient-based improvement to the parameterized acting policy. We investigate the convergence of this approach using the popular analysis scheme of generalized Policy Iteration in the finite-horizon domain. Empirically, incorporating value-improvement into the popular off-policy actor-critic algorithms TD3 and SAC significantly improves or matches performance over their respective baselines, across different environments from the DeepMind continuous control domain, with negligible compute and implementation cost.

翻译：为学习决策问题的近似最优行动策略，现代行动者-评论者算法依赖深度神经网络参数化行动策略，并利用贪心化算子迭代改进策略。对深度神经网络的依赖意味着采用基于梯度的改进方式，其每一步的改进幅度远小于Q-learning算法所用贪心更新等更激进的算子所能实现的改进。另一方面，策略的缓慢更新也有利于学习过程的稳定性，从而形成贪心化与稳定性之间的权衡。为更好地处理这一权衡，我们提出将行动策略与评论者评估的策略解耦。这使得智能体能够以更激进的更新方式单独改进评论者策略（例如价值改进），同时保持对参数化行动策略基于梯度的缓慢改进。我们采用有限时域中广义策略迭代的经典分析框架，对此方法的收敛性进行了理论探讨。实证研究表明，在深度强化学习连续控制领域的多种环境中，将价值改进机制融入主流离轨行动者-评论者算法TD3和SAC后，其性能较基线算法均有显著提升或持平，且仅需可忽略的计算与实现成本。