Most entropy measures depend on the spread of the probability distribution over the sample space $\mathcal{X}$, and the maximum entropy achievable scales proportionately with the sample space cardinality $|\mathcal{X}|$. For a finite $|\mathcal{X}|$, this yields robust entropy measures which satisfy many important properties, such as invariance to bijections, while the same is not true for continuous spaces (where $|\mathcal{X}|=\infty$). Furthermore, since $\mathbb{R}$ and $\mathbb{R}^d$ ($d\in \mathbb{Z}^+$) have the same cardinality (from Cantor's correspondence argument), cardinality-dependent entropy measures cannot encode the data dimensionality. In this work, we question the role of cardinality and distribution spread in defining entropy measures for continuous spaces, which can undergo multiple rounds of transformations and distortions, e.g., in neural networks. We find that the average value of the local intrinsic dimension of a distribution, denoted as ID-Entropy, can serve as a robust entropy measure for continuous spaces, while capturing the data dimensionality. We find that ID-Entropy satisfies many desirable properties and can be extended to conditional entropy, joint entropy and mutual-information variants. ID-Entropy also yields new information bottleneck principles and also links to causality. In the context of deep learning, for feedforward architectures, we show, theoretically and empirically, that the ID-Entropy of a hidden layer directly controls the generalization gap for both classifiers and auto-encoders, when the target function is Lipschitz continuous. Our work primarily shows that, for continuous spaces, taking a structural rather than a statistical approach yields entropy measures which preserve intrinsic data dimensionality, while being relevant for studying various architectures.

翻译：大多数熵度量取决于概率分布在样本空间$\mathcal{X}$上的扩散程度，且可达最大熵与样本空间基数$|\mathcal{X}|$成比例增长。对于有限$|\mathcal{X}|$，这产生了满足双射不变性等重要性质的鲁棒熵度量，但连续空间（其中$|\mathcal{X}|=\infty$）则不然。进一步，由于$\mathbb{R}$和$\mathbb{R}^d$（$d\in \mathbb{Z}^+$）具有相同基数（根据康托尔对应论证），依赖基数的熵度量无法编码数据维数。本文质疑了基数与分布扩散在定义连续空间熵度量中的作用——此类空间可能经历神经网络等多轮变换与扭曲。我们发现，分布的局部本征维数均值（称为ID-熵）可作为连续空间的鲁棒熵度量，同时捕捉数据维数。研究表明ID-熵满足诸多理想性质，并可推广至条件熵、联合熵及互信息变体。ID-熵还衍生出新的信息瓶颈原理，并与因果关系相关联。在深度学习背景下，对于前馈架构，我们从理论与实证两方面证明：当目标函数Lipschitz连续时，隐藏层的ID-熵直接控制分类器与自编码器的泛化差距。本研究核心表明：对于连续空间，采用结构方法而非统计方法可得到保持数据本征维数的熵度量，且对研究各类架构具有相关性。