Recently, it has been shown that for offline deep reinforcement learning (DRL), pre-training Decision Transformer with a large language corpus can improve downstream performance (Reid et al., 2022). A natural question to ask is whether this performance gain can only be achieved with language pre-training, or can be achieved with simpler pre-training schemes which do not involve language. In this paper, we first show that language is not essential for improved performance, and indeed pre-training with synthetic IID data for a small number of updates can match the performance gains from pre-training with a large language corpus; moreover, pre-training with data generated by a one-step Markov chain can further improve the performance. Inspired by these experimental results, we then consider pre-training Conservative Q-Learning (CQL), a popular offline DRL algorithm, which is Q-learning-based and typically employs a Multi-Layer Perceptron (MLP) backbone. Surprisingly, pre-training with simple synthetic data for a small number of updates can also improve CQL, providing consistent performance improvement on D4RL Gym locomotion datasets. The results of this paper not only illustrate the importance of pre-training for offline DRL but also show that the pre-training data can be synthetic and generated with remarkably simple mechanisms.

翻译：近期研究表明，在离线深度强化学习（DRL）中，使用大规模语言语料库对Decision Transformer进行预训练可提升下游任务性能（Reid等人，2022）。这自然引出一个问题：这种性能增益是否仅能通过语言预训练实现，抑或可通过不涉及语言的更简单预训练方案达成？本文首先证明语言并非提升性能的必要条件：使用合成独立同分布数据进行少量更新步数的预训练，即可达到与大规模语言语料库预训练相当的性能增益；而使用一步马尔可夫链生成的数据进行预训练甚至能进一步提升性能。受这些实验结果启发，我们进一步对保守Q学习（CQL）——一种基于Q学习、通常采用多层感知机（MLP）骨干的流行离线DRL算法——进行预训练研究。令人惊讶的是，即使使用简单合成数据进行少量更新步数的预训练，同样能提升CQL的性能，在D4RL Gym运动控制数据集中展现出稳定的性能改进。本文结果不仅说明了预训练对离线DRL的重要性，同时揭示了预训练数据完全可以通过极其简单的机制合成生成。