Offline reinforcement learning (RL) learns effective policies from a static target dataset. The performance of state-of-the-art offline RL algorithms notwithstanding, it relies on the quality and size of the target dataset and it degrades if limited samples in the target dataset are available, which is often the case in real-world applications. To address this issue, domain adaptation that leverages auxiliary samples from related source datasets (such as simulators) can be beneficial. However, establishing the optimal way to trade off the source and target datasets while ensuring provably theoretical guarantees remains an open challenge. To the best of our knowledge, this paper proposes the first framework that theoretically explores the impact of the weights assigned to each dataset on the performance of offline RL. In particular, we establish performance bounds and the existence of an optimal weight, which can be computed in closed form under simplifying assumptions. We also provide algorithmic guarantees in terms of convergence to a neighborhood of the optimum. Notably, these results depend on the quality of the source dataset and the number of samples from the target dataset. Our empirical results on the well-known Procgen benchmark substantiate our theoretical contributions.

翻译：离线强化学习（RL）从静态目标数据集中学习有效策略。尽管现有最先进的离线RL算法表现优异，但其性能仍依赖于目标数据集的质量与规模，当目标数据集样本有限时性能会下降——这在实际应用中极为常见。为解决此问题，利用来自相关源数据集（如模拟器）的辅助样本进行领域自适应具有显著优势。然而，如何在保证可证明理论性能的前提下，权衡源数据集与目标数据集的最优利用方式，仍是亟待解决的开放性问题。据我们所知，本文首次提出从理论层面探讨数据集权重分配对离线RL性能影响的框架。具体而言，我们建立了性能边界并证明了最优权重的存在性，该权重在简化假设下可通过闭式解计算。我们还提供了算法收敛至最优解邻域的保证性理论。值得注意的是，这些结论取决于源数据集的质量与目标数据集的样本数量。我们在经典Procgen基准测试上的实证结果有力支撑了理论贡献。