Reinforcement learning (RL) agents can leverage batches of previously collected data to extract a reasonable control policy. An emerging issue in this offline RL setting, however, is that the bootstrapping update underlying many of our methods suffers from insufficient action-coverage: standard max operator may select a maximal action that has not been seen in the dataset. Bootstrapping from these inaccurate values can lead to overestimation and even divergence. There are a growing number of methods that attempt to approximate an \emph{in-sample} max, that only uses actions well-covered by the dataset. We highlight a simple fact: it is more straightforward to approximate an in-sample \emph{softmax} using only actions in the dataset. We show that policy iteration based on the in-sample softmax converges, and that for decreasing temperatures it approaches the in-sample max. We derive an In-Sample Actor-Critic (AC), using this in-sample softmax, and show that it is consistently better or comparable to existing offline RL methods, and is also well-suited to fine-tuning.

翻译：强化学习（Reinforcement Learning, RL）智能体可利用先前收集的数据批次提取合理控制策略。然而在离线强化学习场景中，许多方法所依赖的bootstrapping更新面临动作覆盖不足的问题：标准max算子可能选择数据集未覆盖的最大动作。基于这些不准确值的bootstrapping会导致过高估计甚至发散。现有越来越多方法试图近似仅使用数据集充分覆盖动作的样本内max（in-sample max）。我们强调一个简单事实：仅使用数据集中的动作来近似样本内softmax更为直观。我们证明基于样本内softmax的策略迭代具有收敛性，且随着温度降低会趋近样本内max。基于该样本内softmax，我们推导出样本内演员-评论家算法（In-Sample Actor-Critic, AC），实验表明该方法持续优于或可比肩现有离线强化学习方法，并特别适用于微调场景。