Model-based methods have recently shown great potential for off-policy evaluation (OPE); offline trajectories induced by behavioral policies are fitted to transitions of Markov decision processes (MDPs), which are used to rollout simulated trajectories and estimate the performance of policies. Model-based OPE methods face two key challenges. First, as offline trajectories are usually fixed, they tend to cover limited state and action space. Second, the performance of model-based methods can be sensitive to the initialization of their parameters. In this work, we propose the variational latent branching model (VLBM) to learn the transition function of MDPs by formulating the environmental dynamics as a compact latent space, from which the next states and rewards are then sampled. Specifically, VLBM leverages and extends the variational inference framework with the recurrent state alignment (RSA), which is designed to capture as much information underlying the limited training data, by smoothing out the information flow between the variational (encoding) and generative (decoding) part of VLBM. Moreover, we also introduce the branching architecture to improve the model's robustness against randomly initialized model weights. The effectiveness of the VLBM is evaluated on the deep OPE (DOPE) benchmark, from which the training trajectories are designed to result in varied coverage of the state-action space. We show that the VLBM outperforms existing state-of-the-art OPE methods in general.

翻译：基于模型的方法近年来在离线策略评估中展现出巨大潜力；由行为策略生成的离线轨迹被拟合到马尔可夫决策过程的转移函数上，这些函数用于生成模拟轨迹并评估策略性能。基于模型的离线策略评估方法面临两个关键挑战。首先，由于离线轨迹通常是固定的，它们往往覆盖有限的状态和动作空间。其次，基于模型的方法的性能可能对参数初始化敏感。在这项工作中，我们提出了变分潜在分支模型（VLBM），通过将环境动态建模为紧凑的潜在空间（从中采样下一状态和奖励），来学习马尔可夫决策过程的转移函数。具体来说，VLBM利用并扩展了带有循环状态对齐（RSA）的变分推断框架，通过平滑VLBM中变分编码与生成解码部分之间的信息流，旨在从有限的训练数据中捕获尽可能多的信息。此外，我们还引入了分支架构，以提高模型对随机初始化模型权重的鲁棒性。我们在深度离线策略评估（DOPE）基准上评估了VLBM的有效性，该基准的训练轨迹设计为导致状态-动作空间的不同覆盖程度。我们证明，VLBM在总体上优于现有的最先进离线策略评估方法。