Model-based methods have recently shown great potential for off-policy evaluation (OPE); offline trajectories induced by behavioral policies are fitted to transitions of Markov decision processes (MDPs), which are used to rollout simulated trajectories and estimate the performance of policies. Model-based OPE methods face two key challenges. First, as offline trajectories are usually fixed, they tend to cover limited state and action space. Second, the performance of model-based methods can be sensitive to the initialization of their parameters. In this work, we propose the variational latent branching model (VLBM) to learn the transition function of MDPs by formulating the environmental dynamics as a compact latent space, from which the next states and rewards are then sampled. Specifically, VLBM leverages and extends the variational inference framework with the recurrent state alignment (RSA), which is designed to capture as much information underlying the limited training data, by smoothing out the information flow between the variational (encoding) and generative (decoding) part of VLBM. Moreover, we also introduce the branching architecture to improve the model's robustness against randomly initialized model weights. The effectiveness of the VLBM is evaluated on the deep OPE (DOPE) benchmark, from which the training trajectories are designed to result in varied coverage of the state-action space. We show that the VLBM outperforms existing state-of-the-art OPE methods in general.

翻译：基于模型的方法近年来在离策略评估（OPE）中展现出巨大潜力：由行为策略生成的离线轨迹被拟合到马尔可夫决策过程（MDP）的转移函数中，进而用于生成模拟轨迹并评估策略性能。基于模型的OPE方法面临两个关键挑战：首先，离线轨迹通常固定不变，导致其覆盖的状态和动作空间有限；其次，此类方法的性能对参数初始化较为敏感。本文提出变分潜在分支模型（VLBM），通过将环境动态建模为紧凑的潜在空间来学习MDP的转移函数，并在此空间中采样下一状态及奖励。具体而言，VLBM利用并扩展了变分推断框架，结合循环状态对齐（RSA）机制，通过平滑VLBM变分（编码）部分与生成（解码）部分之间的信息流，捕获有限训练数据中蕴含的尽可能多的信息。此外，我们引入分支架构以提升模型对随机初始化权重的鲁棒性。在深度OPE（DOPE）基准上的评估实验表明，该基准的训练轨迹被设计为覆盖不同大小的状态-动作空间。实验结果显示，VLBM整体上优于现有最先进的OPE方法。