We study the problem of overcoming exponential sample complexity in differential entropy estimation under Gaussian convolutions. Specifically, we consider the estimation of the differential entropy $h(X+Z)$ via $n$ independently and identically distributed samples of $X$, where $X$ and $Z$ are independent $D$-dimensional random variables with $X$ sub-Gaussian with bounded second moment and $Z\sim\mathcal{N}(0,\sigma^2I_D)$. Under the absolute-error loss, the above problem has a parametric estimation rate of $\frac{c^D}{\sqrt{n}}$, which is exponential in data dimension $D$ and often problematic for applications. We overcome this exponential sample complexity by projecting $X$ to a low-dimensional space via principal component analysis (PCA) before the entropy estimation, and show that the asymptotic error overhead vanishes as the unexplained variance of the PCA vanishes. This implies near-optimal performance for inherently low-dimensional structures embedded in high-dimensional spaces, including hidden-layer outputs of deep neural networks (DNN), which can be used to estimate mutual information (MI) in DNNs. We provide numerical results verifying the performance of our PCA approach on Gaussian and spiral data. We also apply our method to analysis of information flow through neural network layers (c.f. information bottleneck), with results measuring mutual information in a noisy fully connected network and a noisy convolutional neural network (CNN) for MNIST classification.

翻译：我们研究在高斯卷积下克服微分熵估计中指数级样本复杂性的问题。具体考虑通过$X$的$n$个独立同分布样本估计微分熵$h(X+Z)$，其中$X$与$Z$为独立$D$维随机变量，$X$为具有有界二阶矩的次高斯分布，$Z\sim\mathcal{N}(0,\sigma^2I_D)$。在绝对误差损失下，该问题的参数估计速率为$\frac{c^D}{\sqrt{n}}$，随数据维数$D$呈指数增长，在实际应用中常引发问题。我们通过在熵估计前对$X$进行主成分分析(PCA)将其投影至低维空间来克服该指数样本复杂性，并证明当PCA未解释方差趋于零时渐近误差开销随之消失。这表明该方法对嵌入高维空间中的本质低维结构（包括深度神经网络(DNN)的隐藏层输出）具有近似最优性能，可用于估计DNN中的互信息(MI)。我们通过高斯数据和螺旋数据上的数值实验验证了PCA方法的性能。同时将该方法应用于神经网络层间信息流分析（参见信息瓶颈理论），在含噪全连接网络和用于MNIST分类的含噪卷积神经网络(CNN)中实现了互信息的测量。