Most entropy measures depend on the spread of the probability distribution over the sample space X, and the maximum entropy achievable scales proportionately with the sample space cardinality |X|. For a finite |X|, this yields robust entropy measures which satisfy many important properties, such as invariance to bijections, while the same is not true for continuous spaces (where |X|=infinity). Furthermore, since R and R^d (d in Z+) have the same cardinality (from Cantor's correspondence argument), cardinality-dependent entropy measures cannot encode the data dimensionality. In this work, we question the role of cardinality and distribution spread in defining entropy measures for continuous spaces, which can undergo multiple rounds of transformations and distortions, e.g., in neural networks. We find that the average value of the local intrinsic dimension of a distribution, denoted as ID-Entropy, can serve as a robust entropy measure for continuous spaces, while capturing the data dimensionality. We find that ID-Entropy satisfies many desirable properties and can be extended to conditional entropy, joint entropy and mutual-information variants. ID-Entropy also yields new information bottleneck principles and also links to causality. In the context of deep learning, for feedforward architectures, we show, theoretically and empirically, that the ID-Entropy of a hidden layer directly controls the generalization gap for both classifiers and auto-encoders, when the target function is Lipschitz continuous. Our work primarily shows that, for continuous spaces, taking a structural rather than a statistical approach yields entropy measures which preserve intrinsic data dimensionality, while being relevant for studying various architectures.

翻译：大多数熵度量依赖于样本空间X上概率分布的扩展程度，且最大可达到的熵与样本空间基数|X|成正比。对于有限|X|，这能产生满足许多重要性质（如对双射不变性）的鲁棒熵度量，但在连续空间（|X|=无穷）中却不成立。此外，由于R和R^d（d为正整数）具有相同基数（根据康托尔对应论证），依赖于基数的熵度量无法编码数据维度。本研究质疑了基数与分布扩展在定义连续空间熵度量中的作用——这类空间可能经历多轮变换与扭曲（例如在神经网络中）。我们发现，分布局部固有维度的平均值（记作ID-熵）可作为连续空间的鲁棒熵度量，同时能捕获数据维度。ID-熵满足众多理想性质，并可扩展至条件熵、联合熵及互信息变体，还能衍生出新的信息瓶颈原理并与因果关系相关联。在深度学习背景下，针对前馈架构，我们从理论和实验上证明：当目标函数为利普希茨连续时，隐藏层的ID-熵直接控制分类器和自编码器的泛化差距。本工作主要表明，在连续空间中，采用结构性方法而非统计性方法，能生成保留数据固有维度的熵度量，进而适用于各类架构研究。