Inference for Variational Autoencoders (VAEs) consists of learning two models: (1) a generative model, which transforms a simple distribution over a latent space into the distribution over observed data, and (2) an inference model, which approximates the posterior of the latent codes given data. The two components are learned jointly via a lower bound to the generative model's log marginal likelihood. In early phases of joint training, the inference model poorly approximates the latent code posteriors. Recent work showed that this leads optimization to get stuck in local optima, negatively impacting the learned generative model. As such, recent work suggests ensuring a high-quality inference model via iterative training: maximizing the objective function relative to the inference model before every update to the generative model. Unfortunately, iterative training is inefficient, requiring heuristic criteria for reverting from iterative to joint training for speed. Here, we suggest an inference method that trains the generative and inference models independently. It approximates the posterior of the true model a priori; fixing this posterior approximation, we then maximize the lower bound relative to only the generative model. By conventional wisdom, this approach should rely on the true prior and likelihood of the true model to approximate its posterior (which are unknown). However, we show that we can compute a deterministic, model-agnostic posterior approximation (MAPA) of the true model's posterior. We then use MAPA to develop a proof-of-concept inference method. We present preliminary results on low-dimensional synthetic data that (1) MAPA captures the trend of the true posterior, and (2) our MAPA-based inference performs better density estimation with less computation than baselines. Lastly, we present a roadmap for scaling the MAPA-based inference method to high-dimensional data.

翻译：变分自编码器（VAEs）的推理过程包含两个模型的学习：（1）生成模型，将隐空间上的简单分布变换为观测数据上的分布；（2）推理模型，近似给定数据时隐编码的后验分布。这两个组件通过生成模型对数边际似然的下界进行联合学习。在联合训练的早期阶段，推理模型对隐编码后验分布的近似效果较差。近期研究表明，这会导致优化陷入局部最优，对生成模型的学习产生负面影响。为此，最新研究建议通过迭代训练确保推理模型的高质量——即在每次更新生成模型之前，最大化关于推理模型的目标函数。然而，迭代训练效率低下，需要依赖启发式准则来决定何时从迭代训练切换回联合训练以提高速度。本文提出一种独立训练生成模型与推理模型的推理方法。该方法先验地近似真实模型的后验分布；固定该后验近似后，仅针对生成模型最大化下界。传统观点认为，此类方法需依赖真实模型未知的先验分布与似然函数来近似其后验。但本文证明，我们能够计算真实模型后验的一种确定性、模型无关的后验近似（MAPA）。进而利用MAPA开发了一种概念验证性的推理方法。在低维合成数据上的初步结果表明：（1）MAPA能捕捉真实后验分布的趋势；（2）基于MAPA的推理方法在密度估计任务中以更少计算量取得了优于基线方法的性能。最后，本文给出了将MAPA推理方法扩展至高维数据的技术路线图。