An important yet underexplored question in the PAC-Bayes literature is how much tightness we lose by restricting the posterior family to factorized Gaussian distributions when optimizing a PAC-Bayes bound. We investigate this issue by estimating data-independent PAC-Bayes bounds using the optimal posteriors, comparing them to bounds obtained using MFVI. Concretely, we (1) sample from the optimal Gibbs posterior using Hamiltonian Monte Carlo, (2) estimate its KL divergence from the prior with thermodynamic integration, and (3) propose three methods to obtain high-probability bounds under different assumptions. Our experiments on the MNIST dataset reveal significant tightness gaps, as much as 5-6\% in some cases.

翻译：PAC-Bayes文献中一个重要但尚未充分探索的问题是：在优化PAC-Bayes界时，将后验族限制为因子化高斯分布会损失多少紧致性。我们通过使用最优后验估计数据无关的PAC-Bayes界，并将其与使用平均场变分推断（MFVI）获得的界进行比较，来研究这一问题。具体而言，我们（1）使用哈密顿蒙特卡洛从最优吉布斯后验中采样，（2）通过热力学积分估计其与先验的KL散度，以及（3）提出三种在不同假设下获得高概率界的方法。我们在MNIST数据集上的实验揭示了显著的紧致性差距，在某些情况下高达5-6%。