We introduce an information-theoretic quantity with similar properties to mutual information that can be estimated from data without making explicit assumptions on the underlying distribution. This quantity is based on a recently proposed matrix-based entropy that uses the eigenvalues of a normalized Gram matrix to compute an estimate of the eigenvalues of an uncentered covariance operator in a reproducing kernel Hilbert space. We show that a difference of matrix-based entropies (DiME) is well suited for problems involving the maximization of mutual information between random variables. While many methods for such tasks can lead to trivial solutions, DiME naturally penalizes such outcomes. We compare DiME to several baseline estimators of mutual information on a toy Gaussian dataset. We provide examples of use cases for DiME, such as latent factor disentanglement and a multiview representation learning problem where DiME is used to learn a shared representation among views with high mutual information.

翻译：我们提出了一种具有与互信息类似性质的信息论量，该量无需对潜在分布做显式假设即可从数据中估计。该量基于近期提出的矩阵熵，通过归一化格拉姆矩阵的特征值，计算再生核希尔伯特空间中非中心协方差算子特征值的估计值。我们证明，矩阵熵差值（DiME）特别适用于涉及随机变量间互信息最大化的问题。尽管此类任务的许多方法可能导致平凡解，但DiME能自然地惩罚此类结果。我们在玩具高斯数据集上将DiME与多种互信息基线估计器进行了比较。我们提供了DiME的应用案例，例如潜在因子解缠和多视角表示学习问题——在该问题中，DiME被用于学习具有高互信息的多视角间共享表示。