Offline reinforcement learning (RL) learns effective policies from a static target dataset. The performance of state-of-the-art offline RL algorithms notwithstanding, it relies on the size of the target dataset, and it degrades if limited samples in the target dataset are available, which is often the case in real-world applications. To address this issue, domain adaptation that leverages auxiliary samples from related source datasets (such as simulators) can be beneficial. However, establishing the optimal way to trade off the limited target dataset and the large-but-biased source dataset while ensuring provably theoretical guarantees remains an open challenge. To the best of our knowledge, this paper proposes the first framework that theoretically explores the impact of the weights assigned to each dataset on the performance of offline RL. In particular, we establish performance bounds and the existence of the optimal weight, which can be computed in closed form under simplifying assumptions. We also provide algorithmic guarantees in terms of convergence to a neighborhood of the optimum. Notably, these results depend on the quality of the source dataset and the number of samples in the target dataset. Our empirical results on the well-known offline Procgen benchmark substantiate the theoretical contributions in this work.

翻译：离线强化学习（RL）从静态目标数据集中学习有效策略。尽管现有最先进的离线RL算法表现出色，但其性能依赖于目标数据集的规模，当目标数据集样本有限时性能会下降，而这在实际应用中十分常见。为解决这一问题，利用来自相关源数据集（如模拟器）的辅助样本进行域适应具有显著优势。然而，如何在有限的目标数据集与大规模但存在偏差的源数据集之间进行最优权衡，同时确保可证明的理论保证，仍是一个开放挑战。据我们所知，本文首次提出了一个理论框架，用于探究不同数据集权重分配对离线RL性能的影响。具体而言，我们建立了性能边界并证明了最优权重的存在性——在简化假设下该权重可通过闭式解计算。我们还提供了算法在收敛至最优解邻域的理论保证。值得注意的是，这些结果取决于源数据集的质量与目标数据集的样本数量。我们在经典离线Procgen基准测试上的实证结果，进一步验证了本工作的理论贡献。