Model-based offline reinforcement learning (RL), which builds a supervised transition model with logging dataset to avoid costly interactions with the online environment, has been a promising approach for offline policy optimization. As the discrepancy between the logging data and online environment may result in a distributional shift problem, many prior works have studied how to build robust transition models conservatively and estimate the model uncertainty accurately. However, the over-conservatism can limit the exploration of the agent, and the uncertainty estimates may be unreliable. In this work, we propose a novel Model-based Offline policy optimization framework with Adversarial Network (MOAN). The key idea is to use adversarial learning to build a transition model with better generalization, where an adversary is introduced to distinguish between in-distribution and out-of-distribution samples. Moreover, the adversary can naturally provide a quantification of the model's uncertainty with theoretical guarantees. Extensive experiments showed that our approach outperforms existing state-of-the-art baselines on widely studied offline RL benchmarks. It can also generate diverse in-distribution samples, and quantify the uncertainty more accurately.

翻译：基于模型的离线强化学习通过利用日志数据集构建有监督的转移模型，避免与在线环境进行高成本的交互，已成为离线策略优化的重要方法。由于日志数据与在线环境之间的差异可能导致分布偏移问题，现有研究主要致力于保守地构建鲁棒转移模型并准确估计模型不确定性。然而，过度保守会限制智能体的探索能力，且不确定性估计可能不可靠。本文提出一种新颖的基于模型的对抗网络离线策略优化框架（MOAN），其核心思想是利用对抗学习构建具有更强泛化能力的转移模型，通过引入对抗器区分分布内样本与分布外样本。此外，该对抗器可天然提供具有理论保证的模型不确定性量化方法。大量实验表明，本方法在广泛研究的离线强化学习基准测试中优于现有最先进基线模型，既能生成多样化的分布内样本，又能更准确地量化不确定性。