Why do deep neural networks (DNNs) benefit from very high dimensional parameter spaces? Their huge parameter complexities vs. stunning performances in practice is all the more intriguing and not explainable using the standard theory of regular models. In this work, we propose a geometrically flavored information-theoretic approach to study this phenomenon. Namely, we introduce the locally varying dimensionality of the parameter space of neural network models by considering the number of significant dimensions of the Fisher information matrix, and model the parameter space as a manifold using the framework of singular semi-Riemannian geometry. We derive model complexity measures which yield short description lengths for deep neural network models based on their singularity analysis thus explaining the good performance of DNNs despite their large number of parameters.

翻译：为什么深度神经网络（DNNs）能从极高维的参数空间中受益？其巨大的参数复杂度与实际应用中惊人的表现之间的矛盾，使得这一现象更加引人深思，且无法用标准正则模型理论加以解释。本文提出一种基于几何视角的信息论方法研究该现象：我们通过考虑费舍尔信息矩阵的有效维度数量，引入神经网络模型参数空间的局部可变维度，并利用奇异半黎曼几何框架将参数空间建模为流形。基于对深度神经网络模型的奇异性分析，我们推导出能产生短描述长度的模型复杂度度量，从而解释了DNNs在拥有大量参数的情况下仍能表现优异的原因。