Offline Reinforcement Learning (RL) has shown promising results in learning a task-specific policy from a fixed dataset. However, successful offline RL often relies heavily on the coverage and quality of the given dataset. In scenarios where the dataset for a specific task is limited, a natural approach is to improve offline RL with datasets from other tasks, namely, to conduct Multi-Task Data Sharing (MTDS). Nevertheless, directly sharing datasets from other tasks exacerbates the distribution shift in offline RL. In this paper, we propose an uncertainty-based MTDS approach that shares the entire dataset without data selection. Given ensemble-based uncertainty quantification, we perform pessimistic value iteration on the shared offline dataset, which provides a unified framework for single- and multi-task offline RL. We further provide theoretical analysis, which shows that the optimality gap of our method is only related to the expected data coverage of the shared dataset, thus resolving the distribution shift issue in data sharing. Empirically, we release an MTDS benchmark and collect datasets from three challenging domains. The experimental results show our algorithm outperforms the previous state-of-the-art methods in challenging MTDS problems. See https://github.com/Baichenjia/UTDS for the datasets and code.

翻译：离线强化学习(Offline RL)在从固定数据集学习特定任务策略方面展现出良好前景。然而，成功的离线强化学习往往严重依赖给定数据集的覆盖范围和质量。当特定任务的数据集有限时，一种自然的做法是利用其他任务的数据集来改进离线强化学习，即进行多任务数据共享(MTDS)。然而，直接共享其他任务的数据集会加剧离线强化学习中的数据分布偏移。本文提出一种基于不确定性的MTDS方法，无需数据选择即可共享完整数据集。基于集成不确定性量化方法，我们在共享离线数据集上执行悲观值迭代，为单任务和多任务离线强化学习提供了统一框架。我们进一步提供理论分析表明，我们方法的最优性间隙仅与共享数据的期望数据覆盖范围相关，从而解决了数据共享中的分布偏移问题。在实验方面，我们发布了一个MTDS基准测试，并收集了三个具有挑战性领域的数据集。实验结果表明，我们的算法在具有挑战性的MTDS问题上优于先前的最先进方法。数据集和代码见https://github.com/Baichenjia/UTDS。