Model-based methods have recently shown great potential for off-policy evaluation (OPE); offline trajectories induced by behavioral policies are fitted to transitions of Markov decision processes (MDPs), which are used to rollout simulated trajectories and estimate the performance of policies. Model-based OPE methods face two key challenges. First, as offline trajectories are usually fixed, they tend to cover limited state and action space. Second, the performance of model-based methods can be sensitive to the initialization of their parameters. In this work, we propose the variational latent branching model (VLBM) to learn the transition function of MDPs by formulating the environmental dynamics as a compact latent space, from which the next states and rewards are then sampled. Specifically, VLBM leverages and extends the variational inference framework with the recurrent state alignment (RSA), which is designed to capture as much information underlying the limited training data, by smoothing out the information flow between the variational (encoding) and generative (decoding) part of VLBM. Moreover, we also introduce the branching architecture to improve the model's robustness against randomly initialized model weights. The effectiveness of the VLBM is evaluated on the deep OPE (DOPE) benchmark, from which the training trajectories are designed to result in varied coverage of the state-action space. We show that the VLBM outperforms existing state-of-the-art OPE methods in general.

翻译：基于模型的方法近年来在离线策略评估（OPE）中展现出巨大潜力：通过将行为策略生成的离线轨迹拟合到马尔可夫决策过程（MDP）的转移函数中，可借助模拟轨迹展开并评估目标策略的性能。基于模型的OPE方法面临两大关键挑战：首先，离线轨迹通常固定不变，导致其对状态与动作空间的覆盖范围有限；其次，此类方法的性能易受参数初始化的敏感性影响。本文提出变分潜在分支模型（VLBM），通过将环境动态建模为紧凑的潜在空间，并从中采样下一状态与奖励，从而学习MDP的转移函数。具体而言，VLBM利用并扩展了变分推断框架，引入循环状态对齐（RSA）机制：通过平滑VLBM变分（编码）部分与生成（解码）部分之间的信息流，最大限度捕捉有限训练数据中的潜在信息。此外，我们引入分支架构以提升模型对随机初始化权重的鲁棒性。在深度OPE（DOPE）基准上评估VLBM的有效性，该基准中的训练轨迹被设计为对状态-动作空间产生不同程度的覆盖。实验表明，VLBM在总体上优于现有最先进的OPE方法。