Estimating the mutual information from samples from a joint distribution is a challenging problem in both science and engineering. In this work, we realize a variational bound that generalizes both discriminative and generative approaches. Using this bound, we propose a hybrid method to mitigate their respective shortcomings. Further, we propose Predictive Quantization (PQ): a simple generative method that can be easily combined with discriminative estimators for minimal computational overhead. Our propositions yield a tighter bound on the information thanks to the reduced variance of the estimator. We test our methods on a challenging task of correlated high-dimensional Gaussian distributions and a stochastic process involving a system of free particles subjected to a fixed energy landscape. Empirical results show that hybrid methods consistently improved mutual information estimates when compared to the corresponding discriminative counterpart.

翻译：从联合分布样本中估计互信息是科学与工程领域中的一个具有挑战性的问题。在本工作中，我们实现了一个统一判别式与生成式方法的变分界。基于该变分界，我们提出了一种混合方法来分别缓解它们的不足。此外，我们提出了预测量化（Predictive Quantization, PQ）：一种简单的生成式方法，可轻松与判别式估计器结合以最小化计算开销。由于估计器方差的降低，我们的方法得出了更紧的信息界。我们在相关高维高斯分布以及涉及固定能量场中自由粒子系统的随机过程等具有挑战性的任务上测试了所提方法。实验结果表明，与对应的判别式方法相比，混合方法能够持续改进互信息估计。