We investigate the complexity of deep neural networks through the lens of functional equivalence, which posits that different parameterizations can yield the same network function. Leveraging the equivalence property, we present a novel bound on the covering number for deep neural networks, which reveals that the complexity of neural networks can be reduced. Additionally, we demonstrate that functional equivalence benefits optimization, as overparameterized networks tend to be easier to train since increasing network width leads to a diminishing volume of the effective parameter space. These findings can offer valuable insights into the phenomenon of overparameterization and have implications for understanding generalization and optimization in deep learning.

翻译：我们通过功能等价性的视角研究深度神经网络的复杂性，该理论认为不同参数化方式可能产生相同的网络函数。利用等价性特性，我们提出了深度神经网络覆盖数的一个新界，揭示了神经网络的复杂性可能降低。此外，我们证明功能等价性有利于优化，因为过参数化网络往往更易训练——随着网络宽度增加，有效参数空间的体积会趋于缩小。这些发现可为过参数化现象提供重要见解，并对理解深度学习中的泛化与优化具有启示意义。