Recent advances in flow-based offline reinforcement learning (RL) have achieved strong performance by parameterizing policies via flow matching. However, they still face critical trade-offs among expressiveness, optimality, and efficiency. In particular, existing flow policies interpret the $L_2$ regularization as an upper bound of the 2-Wasserstein distance ($W_2$), which can be problematic in offline settings. This issue stems from a fundamental geometric mismatch: the behavioral policy manifold is inherently anisotropic, whereas the $L_2$ (or upper bound of $W_2$) regularization is isotropic and density-insensitive, leading to systematically misaligned optimization directions. To address this, we revisit offline RL from a geometric perspective and show that policy refinement can be formulated as a local transport map: an initial flow policy augmented by a residual displacement. By analyzing the induced density transformation, we derive a local quadratic approximation of the KL-constrained objective governed by the Fisher information matrix, enabling a tractable anisotropic optimization formulation. By leveraging the score function embedded in the flow velocity, we obtain a corresponding quadratic constraint for efficient optimization. Our results reveal that the optimality gap in prior methods arises from their isotropic approximation. In contrast, our framework achieves a controllable approximation error within a provable neighborhood of the optimal solution. Extensive experiments demonstrate state-of-the-art performance across diverse offline RL benchmarks. See project page: https://github.com/ARC0127/Fisher-Decorator.

翻译：近期基于流的离线强化学习方法通过流匹配参数化策略取得了优异性能。然而，此类方法在表达能力、最优性与效率之间仍存在关键权衡。特别地，现有流策略将$L_2$正则化解释为2-Wasserstein距离($W_2$)的上界，这在离线场景中可能引发问题。该问题的根源在于根本性的几何不匹配：行为策略流形本身是各向异性的，而$L_2$（或$W_2$上界）正则化却是各向同性且密度不敏感的，导致优化方向系统性偏离。为此，我们从几何视角重新审视离线强化学习，证明策略优化可表述为局部传输映射：通过残差位移增强初始流策略。通过分析诱导密度变换，我们推导出由Fisher信息矩阵控制的KL约束目标局部二次近似，从而建立可解的各向异性优化框架。通过利用流速度中嵌入的得分函数，我们获得对应的二次约束以实现高效优化。理论分析表明，现有方法的最优性差距源于其各向同性近似，而我们的框架能在最优解的可证明邻域内实现可控近似误差。在多种离线强化学习基准上的大量实验验证了其最先进性能。项目页面：https://github.com/ARC0127/Fisher-Decorator