Variational inference with Gaussian mixture models (GMMs) enables learning of highly tractable yet multi-modal approximations of intractable target distributions with up to a few hundred dimensions. The two currently most effective methods for GMM-based variational inference, VIPS and iBayes-GMM, both employ independent natural gradient updates for the individual components and their weights. We show for the first time, that their derived updates are equivalent, although their practical implementations and theoretical guarantees differ. We identify several design choices that distinguish both approaches, namely with respect to sample selection, natural gradient estimation, stepsize adaptation, and whether trust regions are enforced or the number of components adapted. We argue that for both approaches, the quality of the learned approximations can heavily suffer from the respective design choices: By updating the individual components using samples from the mixture model, iBayes-GMM often fails to produce meaningful updates to low-weight components, and by using a zero-order method for estimating the natural gradient, VIPS scales badly to higher-dimensional problems. Furthermore, we show that information-geometric trust-regions (used by VIPS) are effective even when using first-order natural gradient estimates, and often outperform the improved Bayesian learning rule (iBLR) update used by iBayes-GMM. We systematically evaluate the effects of design choices and show that a hybrid approach significantly outperforms both prior works. Along with this work, we publish our highly modular and efficient implementation for natural gradient variational inference with Gaussian mixture models, which supports 432 different combinations of design choices, facilitates the reproduction of all our experiments, and may prove valuable for the practitioner.

翻译：高斯混合模型（GMM）的变分推断能够学习高度可处理且多模态的近似分布，这些分布可逼近维度高达数百的难以处理的目标分布。目前两种最有效的基于GMM的变分推断方法——VIPS和iBayes-GMM——均对单个分量及其权重采用独立的自然梯度更新。我们首次证明，尽管两者的实际实现方式和理论保证不同，但其推导出的更新公式是等价的。我们识别出区分这两种方法的若干设计选择，具体涉及样本选择、自然梯度估计、步长调整，以及是否强制执行置信域或自适应调整分量数量。我们认为，这两种方法的学习近似质量会严重受各自设计选择影响：iBayes-GMM通过从混合模型中采样更新单个分量，常无法对低权重分量生成有意义的更新；而VIPS使用零阶方法估计自然梯度，在处理高维问题时扩展性较差。此外，我们证明信息几何置信域（VIPS所使用）即使在使用一阶自然梯度估计时依然有效，并且通常优于iBayes-GMM使用的改进贝叶斯学习规则（iBLR）更新。我们系统评估了设计选择的影响，并表明混合方法显著优于先前两种工作。伴随本研究，我们发布了基于高斯混合模型的自然梯度变分推断的高度模块化、高效实现，该实现支持432种不同的设计选择组合，便于复现所有实验，并可为实践者提供重要参考。