Existing theoretical studies on offline reinforcement learning (RL) mostly consider a dataset sampled directly from the target task. In practice, however, data often come from several heterogeneous but related sources. Motivated by this gap, this work aims at rigorously understanding offline RL with multiple datasets that are collected from randomly perturbed versions of the target task instead of from itself. An information-theoretic lower bound is derived, which reveals a necessary requirement on the number of involved sources in addition to that on the number of data samples. Then, a novel HetPEVI algorithm is proposed, which simultaneously considers the sample uncertainties from a finite number of data samples per data source and the source uncertainties due to a finite number of available data sources. Theoretical analyses demonstrate that HetPEVI can solve the target task as long as the data sources collectively provide a good data coverage. Moreover, HetPEVI is demonstrated to be optimal up to a polynomial factor of the horizon length. Finally, the study is extended to offline Markov games and offline robust RL, which demonstrates the generality of the proposed designs and theoretical analyses.

翻译：现有关于离线强化学习的理论研究大多考虑直接从目标任务采样的数据集。然而在实践中，数据往往来自多个异质但相关的来源。针对这一差距，本文旨在严格理解使用从目标任务的随机扰动版本（而非其本身）收集的多个数据集进行离线强化学习。我们推导出一个信息论下界，该下界揭示了除了数据样本数量要求外，对涉及的数据源数量也存在必要约束。随后提出了一种新颖的HetPEVI算法，该算法同时考虑了每个数据源有限数据样本带来的样本不确定性，以及有限可用数据源数量导致的源不确定性。理论分析表明，只要数据源共同提供良好的数据覆盖，HetPEVI即可解决目标任务。此外，HetPEVI被证明在最多一个水平长度多项式因子范围内是最优的。最后，我们将研究扩展到离线马尔可夫博弈和离线鲁棒强化学习，证明了所提出设计与理论分析的普适性。