Offline estimation of the dynamical model of a Markov Decision Process (MDP) is a non-trivial task that greatly depends on the data available in the learning phase. Sometimes the dynamics of the model is invariant with respect to some transformations of the current state and action. Recent works showed that an expert-guided pipeline relying on Density Estimation methods as Deep Neural Network based Normalizing Flows effectively detects this structure in deterministic environments, both categorical and continuous-valued. The acquired knowledge can be exploited to augment the original data set, leading eventually to a reduction in the distributional shift between the true and the learned model. Such data augmentation technique can be exploited as a preliminary process to be executed before adopting an Offline Reinforcement Learning architecture, increasing its performance. In this work we extend the paradigm to also tackle non-deterministic MDPs, in particular, 1) we propose a detection threshold in categorical environments based on statistical distances, and 2) we show that the former results lead to a performance improvement when solving the learned MDP and then applying the optimized policy in the real environment.

翻译：马尔可夫决策过程（MDP）动力模型的离线估计是一项非平凡任务，其效果高度依赖于学习阶段可用的数据。有时，模型的动力学对当前状态和动作的某些变换具有不变性。最新研究表明，依赖深度神经网络归一化流等密度估计方法的专家引导流程，能够有效检测确定性环境（包括类别型与连续型环境）中的这种结构。所获取的知识可用于扩充原始数据集，最终减少真实模型与学习模型之间的分布偏移。此类数据增强技术可作为采用离线强化学习架构前的预处理步骤，从而提升其性能。在本工作中，我们将该范式扩展至非确定性MDP，具体包括：1）提出基于统计距离的类别型环境检测阈值；2）证明在求解学习到的MDP并将优化策略应用于真实环境时，前述结果能带来性能提升。