Estimating the mutual information from samples from a joint distribution is a challenging problem in both science and engineering. In this work, we realize a variational bound that generalizes both discriminative and generative approaches. Using this bound, we propose a hybrid method to mitigate their respective shortcomings. Further, we propose Predictive Quantization (PQ): a simple generative method that can be easily combined with discriminative estimators for minimal computational overhead. Our propositions yield a tighter bound on the information thanks to the reduced variance of the estimator. We test our methods on a challenging task of correlated high-dimensional Gaussian distributions and a stochastic process involving a system of free particles subjected to a fixed energy landscape. Empirical results show that hybrid methods consistently improved mutual information estimates when compared to the corresponding discriminative counterpart.

翻译：从联合分布样本中估计互信息是科学和工程领域中的一个具有挑战性的问题。在本工作中，我们实现了一个能够统一推广判别式和生成式方法的变分界。利用该界，我们提出了一种混合方法以缓解它们各自的缺点。进一步地，我们提出了预测量化（PQ）：一种简单的生成式方法，可轻松与判别式估计器结合，且计算开销极低。得益于估计器方差的减小，我们的方法在信息度量上得到了更紧的界。我们在相关的高维高斯分布这一具有挑战性的任务，以及涉及受固定能量景观影响的自由粒子系统的随机过程上测试了我们的方法。实验结果表明，与对应的判别式方法相比，混合方法一致地改进了互信息估计。