A key theme in the past decade has been that when large neural networks and large datasets combine they can produce remarkable results. In deep reinforcement learning (RL), this paradigm is commonly made possible through experience replay, whereby a dataset of past experiences is used to train a policy or value function. However, unlike in supervised or self-supervised learning, an RL agent has to collect its own data, which is often limited. Thus, it is challenging to reap the benefits of deep learning, and even small neural networks can overfit at the start of training. In this work, we leverage the tremendous recent progress in generative modeling and propose Synthetic Experience Replay (SynthER), a diffusion-based approach to arbitrarily upsample an agent's collected experience. We show that SynthER is an effective method for training RL agents across offline and online settings. In offline settings, we observe drastic improvements both when upsampling small offline datasets and when training larger networks with additional synthetic data. Furthermore, SynthER enables online agents to train with a much higher update-to-data ratio than before, leading to a large increase in sample efficiency, without any algorithmic changes. We believe that synthetic training data could open the door to realizing the full potential of deep learning for replay-based RL algorithms from limited data.

翻译：过去十年的一个关键主题是，当大型神经网络与大型数据集相结合时，能够产生显著的结果。在深度强化学习（RL）中，这一范式通常通过经验回放实现，即利用过去经验的数据集来训练策略或价值函数。然而，与监督学习或自监督学习不同，RL智能体需要自行收集数据，而这类数据往往有限。因此，要获得深度学习的收益颇具挑战性，甚至小型神经网络在训练初期也可能出现过拟合。在此项工作中，我们利用生成建模领域近期取得的巨大进展，提出了一种基于扩散的方法——合成经验回放（SynthER），用于对智能体收集的经验进行任意倍数的上采样。我们证明，无论是在离线还是在线场景下，SynthER都是训练RL智能体的有效方法。在离线场景中，我们观察到，无论是对小型离线数据集进行上采样，还是利用额外的合成数据训练更大规模的网络，都实现了显著的性能提升。此外，SynthER使得在线智能体能够以比以往高得多的更新-数据比进行训练，从而在无需任何算法改动的情况下大幅提高了样本效率。我们相信，合成训练数据或将为从有限数据中实现基于回放的RL算法充分发挥深度学习的潜力铺平道路。