In this paper, we study the continual learning problem of single-task offline reinforcement learning. In the past, continual reinforcement learning usually only dealt with multitasking, that is, learning multiple related or unrelated tasks in a row, but once each learned task was learned, it was not relearned, but only used in subsequent processes. However, offline reinforcement learning tasks require the continuously learning of multiple different datasets for the same task. Existing algorithms will try their best to achieve the best results in each offline dataset they have learned and the skills of the network will overwrite the high-quality datasets that have been learned after learning the subsequent poor datasets. On the other hand, if too much emphasis is placed on stability, the network will learn the subsequent better dataset after learning the poor offline dataset, and the problem of insufficient plasticity and non-learning will occur. How to design a strategy that can always preserve the best performance for each state in the data that has been learned is a new challenge and the focus of this study. Therefore, this study proposes a new algorithm, called Ensemble Offline Reinforcement Learning Based on Experience Replay, which introduces multiple value networks to learn the same dataset and judge whether the strategy has been learned by the discrete degree of the value network, to improve the performance of the network in single-task offline reinforcement learning.

翻译：本文研究了单任务离线强化学习中的连续学习问题。过去，连续强化学习通常仅处理多任务场景，即连续学习多个相关或不相关的任务，但每个任务一旦学习完成后不再重新学习，仅在后续过程中使用。然而，离线强化学习任务要求针对同一任务持续学习多个不同数据集。现有算法会尽力在每个已学习的离线数据集上取得最佳结果，但网络技能会在学习后续低质量数据集后覆盖已学习的高质量数据集。另一方面，若过度强调稳定性，网络在学习低质量离线数据集后面对后续更优数据集时，会出现可塑性不足与无法学习的问题。如何设计一种策略，在已学习数据中始终为每个状态保留最优性能，成为一项新挑战，也是本研究的焦点。为此，本研究提出了一种新算法——基于经验回放的集成离线强化学习，通过引入多个价值网络学习同一数据集，并利用价值网络的离散程度判断策略是否已被学习，以提升网络在单任务离线强化学习中的性能。