We study the finite-horizon offline reinforcement learning (RL) problem. Since actions at any state can affect next-state distributions, the related distributional shift challenges can make this problem far more statistically complex than offline policy learning for a finite sequence of stochastic contextual bandit environments. We formalize this insight by showing that the statistical hardness of offline RL instances can be measured by estimating the size of actions' impact on next-state distributions. Furthermore, this estimated impact allows us to propagate just enough value function uncertainty from future steps to avoid model exploitation, enabling us to develop algorithms that improve upon traditional pessimistic approaches for offline RL on statistically simple instances. Our approach is supported by theory and simulations.

翻译：我们研究有限时间范围的离线强化学习问题。由于任何状态下的动作都可能影响下一状态的分布，相关的分布偏移挑战使得该问题在统计复杂性上远高于有限序列随机情境赌博机环境下的离线策略学习。我们通过证明离线强化学习实例的统计难度可通过估计动作对下一状态分布影响的大小来衡量，将这一见解形式化。进一步地，这种估计影响使我们能够从未来步骤传播恰好足够的价值函数不确定性以避免模型过度利用，从而开发出在统计简单实例上优于传统悲观方法的离线强化学习算法。我们的方法获得了理论分析与仿真实验的支持。