Offline reinforcement learning (RL) struggles in environments with rich and noisy inputs, where the agent only has access to a fixed dataset without environment interactions. Past works have proposed common workarounds based on the pre-training of state representations, followed by policy training. In this work, we introduce a simple, yet effective approach for learning state representations. Our method, Behavior Prior Representation (BPR), learns state representations with an easy-to-integrate objective based on behavior cloning of the dataset: we first learn a state representation by mimicking actions from the dataset, and then train a policy on top of the fixed representation, using any off-the-shelf Offline RL algorithm. Theoretically, we prove that BPR carries out performance guarantees when integrated into algorithms that have either policy improvement guarantees (conservative algorithms) or produce lower bounds of the policy values (pessimistic algorithms). Empirically, we show that BPR combined with existing state-of-the-art Offline RL algorithms leads to significant improvements across several offline control benchmarks. The code is available at \url{https://github.com/bit1029public/offline_bpr}.

翻译：离线强化学习（Offline RL）在具有丰富且含噪输入的环境中面临挑战，此时智能体仅能访问固定数据集而无法与环境交互。以往的工作提出了基于状态表征预训练后接策略训练的常见解决方案。本文提出了一种简单而有效的状态表征学习方法。我们的方法——行为先验表征（BPR），通过基于数据集行为克隆的易集成目标来学习状态表征：首先通过模仿数据集中的动作学习状态表征，然后基于固定表征使用任意现成的离线强化学习算法训练策略。理论上，我们证明BPR在集成到具有策略改进保证（保守算法）或能产生策略值下界（悲观算法）的算法时，能够提供性能保障。实验上，我们展示了BPR与现有最先进的离线强化学习算法结合，在多个离线控制基准测试中取得了显著改进。代码开源地址：\url{https://github.com/bit1029public/offline_bpr}。