Reinforcement learning (RL) agents can leverage batches of previously collected data to extract a reasonable control policy. An emerging issue in this offline RL setting, however, is that the bootstrapping update underlying many of our methods suffers from insufficient action-coverage: standard max operator may select a maximal action that has not been seen in the dataset. Bootstrapping from these inaccurate values can lead to overestimation and even divergence. There are a growing number of methods that attempt to approximate an \emph{in-sample} max, that only uses actions well-covered by the dataset. We highlight a simple fact: it is more straightforward to approximate an in-sample \emph{softmax} using only actions in the dataset. We show that policy iteration based on the in-sample softmax converges, and that for decreasing temperatures it approaches the in-sample max. We derive an In-Sample Actor-Critic (AC), using this in-sample softmax, and show that it is consistently better or comparable to existing offline RL methods, and is also well-suited to fine-tuning.

翻译：强化学习（RL）智能体可以利用先前收集的数据批次提取合理的控制策略。然而，在这种离线RL设置中，一个新兴问题是支撑许多方法的自举更新遭受动作覆盖不足的困扰：标准最大算子可能选择一个数据集中未曾出现的最大动作。从这些不准确的值进行自举会导致过估计甚至发散。越来越多的方法试图近似一种仅使用数据集充分覆盖动作的样本内最大化。我们强调一个简单事实：仅使用数据集中的动作来近似样本内软最大化更为直接。我们表明，基于样本内软最大化的策略迭代是收敛的，并且对于递减的温度参数，它趋近于样本内最大化。我们利用这种样本内软最大化推导出样本内演员-评论家（AC）算法，并证明它在性能上始终优于或等同于现有离线RL方法，同时特别适合微调场景。