In this work we present a new method for the estimation of Mutual Information (MI) between random variables. Our approach is based on an original interpretation of the Girsanov theorem, which allows us to use score-based diffusion models to estimate the Kullback Leibler divergence between two densities as a difference between their score functions. As a by-product, our method also enables the estimation of the entropy of random variables. Armed with such building blocks, we present a general recipe to measure MI, which unfolds in two directions: one uses conditional diffusion process, whereas the other uses joint diffusion processes that allow simultaneous modelling of two random variables. Our results, which derive from a thorough experimental protocol over all the variants of our approach, indicate that our method is more accurate than the main alternatives from the literature, especially for challenging distributions. Furthermore, our methods pass MI self-consistency tests, including data processing and additivity under independence, which instead are a pain-point of existing methods.

翻译：摘要：本文提出了一种新的随机变量间互信息估计方法。我们的方法基于Girsanov定理的创新性诠释，通过使用基于得分的扩散模型，将两个密度之间的Kullback-Leibler散度估计为其得分函数之差。作为副产品，该方法还可用于估计随机变量的熵。借助这些构建模块，我们提出了通用的互信息测量方案，该方案沿两个方向展开：其一利用条件扩散过程，其二则采用可同时建模两个随机变量的联合扩散过程。经过对所有变体方法的严格实验验证，结果表明我们的方法相比文献中的主流替代方案具有更高精度，尤其适用于具有挑战性的分布场景。此外，我们的方法通过了互信息自洽性检验（包括数据处理不等式和独立性可加性检验），而这些正是现有方法的痛点所在。