In this work we present a new method for the estimation of Mutual Information (MI) between random variables. Our approach is based on an original interpretation of the Girsanov theorem, which allows us to use score-based diffusion models to estimate the Kullback Leibler divergence between two densities as a difference between their score functions. As a by-product, our method also enables the estimation of the entropy of random variables. Armed with such building blocks, we present a general recipe to measure MI, which unfolds in two directions: one uses conditional diffusion process, whereas the other uses joint diffusion processes that allow simultaneous modelling of two random variables. Our results, which derive from a thorough experimental protocol over all the variants of our approach, indicate that our method is more accurate than the main alternatives from the literature, especially for challenging distributions. Furthermore, our methods pass MI self-consistency tests, including data processing and additivity under independence, which instead are a pain-point of existing methods.

翻译：本文提出了一种估计随机变量间互信息（MI）的新方法。该方法基于对Girsanov定理的原创性解释，利用基于分数的扩散模型，将两个密度之间的Kullback-Leibler散度估计为它们得分函数之差。作为副产品，该方法还可用于估计随机变量的熵。基于这些构建模块，我们提出了一套通用的MI测量方案，该方案可沿两个方向展开：其一使用条件扩散过程，其二使用联合扩散过程以实现对两个随机变量的同时建模。通过对所有变体进行严格实验协议验证，结果表明：与文献中的主要替代方法相比，我们的方法在具有挑战性的分布场景中精度更高。此外，我们的方法通过了MI自洽性检验（包括数据处理不等式与独立性可加性），而现有方法在这些方面存在明显不足。