Most entropy measures depend on the spread of the probability distribution over the sample space X, and the maximum entropy achievable scales proportionately with the sample space cardinality |X|. For a finite |X|, this yields robust entropy measures which satisfy many important properties, such as invariance to bijections, while the same is not true for continuous spaces (where |X|=infinity). Furthermore, since R and R^d (d in Z+) have the same cardinality (from Cantor's correspondence argument), cardinality-dependent entropy measures cannot encode the data dimensionality. In this work, we question the role of cardinality and distribution spread in defining entropy measures for continuous spaces, which can undergo multiple rounds of transformations and distortions, e.g., in neural networks. We find that the average value of the local intrinsic dimension of a distribution, denoted as ID-Entropy, can serve as a robust entropy measure for continuous spaces, while capturing the data dimensionality. We find that ID-Entropy satisfies many desirable properties and can be extended to conditional entropy, joint entropy and mutual-information variants. ID-Entropy also yields new information bottleneck principles and also links to causality. In the context of deep learning, for feedforward architectures, we show, theoretically and empirically, that the ID-Entropy of a hidden layer directly controls the generalization gap for both classifiers and auto-encoders, when the target function is Lipschitz continuous. Our work primarily shows that, for continuous spaces, taking a structural rather than a statistical approach yields entropy measures which preserve intrinsic data dimensionality, while being relevant for studying various architectures.

翻译：大多数熵度量依赖于概率分布在样本空间X上的分散程度，且最大可达熵与样本空间基数|X|成正比。对于有限|X|，这能产生满足诸多重要性质（如对双射的不变性）的稳健熵度量，但对于连续空间（|X|=无穷大）则不然。此外，由于R和R^d（d∈Z+）具有相同的基数（根据康托尔对应论证），依赖于基数的熵度量无法编码数据的维度。本文中，我们质疑基数与分布分散程度在定义连续空间熵度量中的作用——这些空间可能经历多轮变换与扭曲（例如在神经网络中）。我们发现，分布局部本征维度的平均值（称为ID-熵）可作为连续空间的稳健熵度量，同时捕获数据维度。研究表明，ID-熵满足许多理想性质，并可扩展至条件熵、联合熵及互信息变体。ID-熵还能导出新的信息瓶颈原理并与因果性相关联。在深度学习背景下，针对前馈架构，我们从理论和实验上证明：当目标函数满足利普希茨连续时，隐藏层的ID-熵直接控制分类器与自编码器的泛化差距。本文主要表明，对于连续空间，采用结构而非统计方法得到的熵度量能保留数据的本征维度，且对研究各类架构具有重要意义。