Stochastic models with global parameters $\bm{\theta}$ and latent variables $\bm{z}$ are common, and variational inference (VI) is popular for their estimation. This paper uses a variational approximation (VA) that comprises a Gaussian with factor covariance matrix for the marginal of $\bm{\theta}$, and the exact conditional posterior of $\bm{z}|\bm{\theta}$. Stochastic optimization for learning the VA only requires generation of $\bm{z}$ from its conditional posterior, while $\bm{\theta}$ is updated using the natural gradient, producing a hybrid VI method. We show that this is a well-defined natural gradient optimization algorithm for the joint posterior of $(\bm{z},\bm{\theta})$. Fast to compute expressions for the Tikhonov damped Fisher information matrix required to compute a stable natural gradient update are derived. We use the approach to estimate probabilistic Bayesian neural networks with random output layer coefficients to allow for heterogeneity. Simulations show that using the natural gradient is more efficient than using the ordinary gradient, and that the approach is faster and more accurate than two leading benchmark natural gradient VI methods. In a financial application we show that accounting for industry level heterogeneity using the deep model improves the accuracy of probabilistic prediction of asset pricing models.

翻译：具有全局参数$\bm{\theta}$和隐变量$\bm{z}$的随机模型十分常见，变分推断（VI）是估计此类模型的常用方法。本文采用一种变分近似（VA），其中包含一个因子协方差矩阵的高斯分布用于$\bm{\theta}$的边缘分布，以及$\bm{z}|\bm{\theta}$的精确条件后验分布。学习VA的随机优化仅需从条件后验中生成$\bm{z}$，同时利用自然梯度更新$\bm{\theta}$，从而产生一种混合VI方法。我们证明这是一种针对$(\bm{z},\bm{\theta})$联合后验的良好定义的自然梯度优化算法。本文推导了计算稳定自然梯度更新所需的Tikhonov阻尼Fisher信息矩阵的快速表达式。我们使用该方法估计具有随机输出层系数的概率贝叶斯神经网络，以允许异质性。仿真表明，使用自然梯度比普通梯度更高效，且该方法比两种领先的基准自然梯度VI方法更快、更准确。在一个金融应用中，我们展示了使用深层模型考虑行业层面异质性可提高资产定价模型概率预测的准确性。