We develop EigenVI, an eigenvalue-based approach for black-box variational inference (BBVI). EigenVI constructs its variational approximations from orthogonal function expansions. For distributions over $\mathbb{R}^D$, the lowest order term in these expansions provides a Gaussian variational approximation, while higher-order terms provide a systematic way to model non-Gaussianity. These approximations are flexible enough to model complex distributions (multimodal, asymmetric), but they are simple enough that one can calculate their low-order moments and draw samples from them. EigenVI can also model other types of random variables (e.g., nonnegative, bounded) by constructing variational approximations from different families of orthogonal functions. Within these families, EigenVI computes the variational approximation that best matches the score function of the target distribution by minimizing a stochastic estimate of the Fisher divergence. Notably, this optimization reduces to solving a minimum eigenvalue problem, so that EigenVI effectively sidesteps the iterative gradient-based optimizations that are required for many other BBVI algorithms. (Gradient-based methods can be sensitive to learning rates, termination criteria, and other tunable hyperparameters.) We use EigenVI to approximate a variety of target distributions, including a benchmark suite of Bayesian models from posteriordb. On these distributions, we find that EigenVI is more accurate than existing methods for Gaussian BBVI.

翻译：我们开发了EigenVI，一种基于特征值的黑盒变分推断（BBVI）方法。EigenVI通过正交函数展开构建其变分近似。对于定义在$\mathbb{R}^D$上的分布，展开式的最低阶项提供高斯变分近似，而高阶项则为建模非高斯性提供系统化途径。此类近似具有足够的灵活性以建模复杂分布（多峰、非对称），同时又足够简洁，能够计算其低阶矩并从中采样。通过采用不同正交函数族构建变分近似，EigenVI还可建模其他类型的随机变量（如非负、有界变量）。在这些函数族中，EigenVI通过最小化Fisher散度的随机估计，计算与目标分布评分函数最匹配的变分近似。值得注意的是，该优化问题可归结为求解最小特征值问题，从而使EigenVI有效规避了其他多数BBVI算法所需的迭代梯度优化过程（基于梯度的方法对学习率、终止准则及其他可调超参数较为敏感）。我们使用EigenVI近似了多种目标分布，包括来自posteriordb的贝叶斯模型基准测试集。实验表明，对于高斯BBVI任务，EigenVI相比现有方法具有更高精度。