Offline reinforcement learning (RL) enables data-efficient and safe policy learning without online exploration, but its performance often degrades under distribution shift. The learned policy may visit out-of-distribution state-action pairs where value estimates and learned dynamics are unreliable. To address policy-induced extrapolation and transition uncertainty in a unified framework, we formulate offline RL as robust policy optimization, treating the transition kernel as a decision variable within an uncertainty set and optimizing the policy against the worst-case dynamics. We propose Robust Regularized Policy Iteration (RRPI), which replaces the intractable max-min bilevel objective with a tractable KL-regularized surrogate and derives an efficient policy iteration procedure based on a robust regularized Bellman operator. We provide theoretical guarantees by showing that the proposed operator is a $γ$-contraction and that iteratively updating the surrogate yields monotonic improvement of the original robust objective with convergence. Experiments on D4RL benchmarks demonstrate that RRPI achieves strong average performance, outperforming recent baselines including percentile-based methods such as PMDB on the majority of environments while remaining competitive on the rest. Moreover, RRPI exhibits robust behavior. The learned $Q$-values decrease in regions with higher epistemic uncertainty, suggesting that the resulting policy avoids unreliable out-of-distribution actions under transition uncertainty.

翻译：离线强化学习（RL）使得无需在线探索即可实现数据高效且安全的策略学习，但其性能在分布偏移下常常会下降。习得的策略可能会访问到分布外的状态-动作对，在这些情况下价值估计和习得的动态模型均不可靠。为了在一个统一框架下解决策略诱导的外推和过渡不确定性，我们将离线RL表述为鲁棒策略优化问题，将转移核视为不确定性集内的一个决策变量，并针对最坏情况的动态优化策略。我们提出了鲁棒正则化策略迭代（RRPI），该方法用一个可处理的KL正则化代理目标替代了难以求解的最大-最小双层目标，并基于一个鲁棒正则化贝尔曼算子推导出了一个高效的策略迭代过程。我们提供了理论保证，证明了所提出的算子是一个$γ$-收缩，并且迭代更新该代理目标能以收敛的方式单调改进原始的鲁棒目标。在D4RL基准测试上的实验表明，RRPI实现了强大的平均性能，在大多数环境中优于包括基于百分位数的方法（如PMDB）在内的近期基线方法，同时在其余环境中保持竞争力。此外，RRPI展现出鲁棒的行为。习得的$Q$值在认知不确定性较高的区域会降低，这表明所得策略在过渡不确定性下避免了不可靠的分布外动作。