Offline reinforcement learning aims to train a policy on a pre-recorded and fixed dataset without any additional environment interactions. There are two major challenges in this setting: (1) extrapolation error caused by approximating the value of state-action pairs not well-covered by the training data and (2) distributional shift between behavior and inference policies. One way to tackle these problems is to induce conservatism - i.e., keeping the learned policies closer to the behavioral ones. To achieve this, we build upon recent works on learning policies in latent action spaces and use a special form of Normalizing Flows for constructing a generative model, which we use as a conservative action encoder. This Normalizing Flows action encoder is pre-trained in a supervised manner on the offline dataset, and then an additional policy model - controller in the latent space - is trained via reinforcement learning. This approach avoids querying actions outside of the training dataset and therefore does not require additional regularization for out-of-dataset actions. We evaluate our method on various locomotion and navigation tasks, demonstrating that our approach outperforms recently proposed algorithms with generative action models on a large portion of datasets.

翻译：离线强化学习旨在基于预录的固定数据集训练策略，无需与环境进行额外交互。该场景面临两大挑战：（1）由训练数据未充分覆盖的状态-动作对价值估计引发的外推误差；（2）行为策略与推理策略之间的分布偏移。解决这些问题的途径之一是引入保守性——即保持学习策略更接近行为策略。为此，我们借鉴了近期在隐动作空间中学习策略的研究，采用一种特殊形式的归一化流构建生成模型，并将其作为保守型动作编码器。该归一化流动作编码器首先在离线数据集上以监督方式进行预训练，随后通过强化学习训练一个额外的策略模型——即隐空间中的控制器。这种方法避免了查询训练数据集之外的动作，因此无需对超出数据集的动作进行额外正则化。我们在多种运动控制与导航任务上评估了该方法，结果表明，在大部分数据集上，我们的方法优于近期提出的基于生成动作模型的算法。