Model-based methods have recently shown great potential for off-policy evaluation (OPE); offline trajectories induced by behavioral policies are fitted to transitions of Markov decision processes (MDPs), which are used to rollout simulated trajectories and estimate the performance of policies. Model-based OPE methods face two key challenges. First, as offline trajectories are usually fixed, they tend to cover limited state and action space. Second, the performance of model-based methods can be sensitive to the initialization of their parameters. In this work, we propose the variational latent branching model (VLBM) to learn the transition function of MDPs by formulating the environmental dynamics as a compact latent space, from which the next states and rewards are then sampled. Specifically, VLBM leverages and extends the variational inference framework with the recurrent state alignment (RSA), which is designed to capture as much information underlying the limited training data, by smoothing out the information flow between the variational (encoding) and generative (decoding) part of VLBM. Moreover, we also introduce the branching architecture to improve the model's robustness against randomly initialized model weights. The effectiveness of the VLBM is evaluated on the deep OPE (DOPE) benchmark, from which the training trajectories are designed to result in varied coverage of the state-action space. We show that the VLBM outperforms existing state-of-the-art OPE methods in general.

翻译：基于模型的方法近期在离线策略评估（OPE）中展现出巨大潜力；行为策略生成的离线轨迹被拟合为马尔可夫决策过程（MDP）的转移，进而用于生成模拟轨迹并评估策略性能。基于模型的OPE方法面临两大关键挑战：首先，由于离线轨迹通常是固定的，其覆盖的状态和动作空间往往有限；其次，模型方法的性能可能对其参数初始化敏感。本文提出变分潜在分支模型（VLBM），通过将环境动态建模为紧凑潜在空间来学习MDP的转移函数，并从中采样后续状态与奖励。具体而言，VLBM利用并扩展了带有循环状态对齐（RSA）的变分推断框架，通过平滑变分（编码）部分与生成（解码）部分之间的信息流，从有限的训练数据中捕获更多潜在信息。此外，我们引入分支架构以提升模型对随机初始化权重的鲁棒性。在深度OPE（DOPE）基准上评估了VLBM的有效性，该基准设计的训练轨迹会导致状态-动作空间覆盖率的差异。实验表明，VLBM总体上优于现有最先进的OPE方法。