Black-box variational inference is widely used in situations where there is no proof that its stochastic optimization succeeds. We suggest this is due to a theoretical gap in existing stochastic optimization proofs: namely the challenge of gradient estimators with unusual noise bounds, and a composite non-smooth objective. For dense Gaussian variational families, we observe that existing gradient estimators based on reparameterization satisfy a quadratic noise bound and give novel convergence guarantees for proximal and projected stochastic gradient descent using this bound. This provides rigorous guarantees that methods similar to those used in practice converge on realistic inference problems.

翻译：黑箱变分推论广泛用于无法证明其随机优化成功的情形。我们认为这源于现有随机优化证明中的理论空白：即梯度估计量具有特殊噪声界和复合非光滑目标函数的挑战。对于稠密高斯变分族，我们观察到基于重参数化的现有梯度估计量满足二次噪声界，并利用该界给出了近端和投影随机梯度下降法的新收敛保证。这为实践中类似方法在现实推理问题上的收敛性提供了严格的理论保障。