We investigate the approximation capabilities of dense neural networks. While universal approximation theorems establish that sufficiently large architectures can approximate arbitrary continuous functions if there are no restrictions on the weight values, we show that dense neural networks do not possess this universality. Our argument is based on a model compression approach, combining the weak regularity lemma with an interpretation of feedforward networks as message passing graph neural networks. We consider ReLU neural networks subject to natural constraints on weights and input and output dimensions, which model a notion of dense connectivity. Within this setting, we demonstrate the existence of Lipschitz continuous functions that cannot be approximated by such networks. This highlights intrinsic limitations of neural networks with dense layers and motivates the use of sparse connectivity as a necessary ingredient for achieving true universality.

翻译：本研究探讨了稠密神经网络的逼近能力。尽管通用逼近定理表明，若对权重值无限制，足够大的网络架构能够逼近任意连续函数，但我们证明稠密神经网络并不具备这种通用性。我们的论证基于模型压缩方法，将弱正则性引理与前馈网络作为消息传递图神经网络的解释相结合。我们考虑受权重及输入输出维度自然约束的ReLU神经网络，这些约束建模了稠密连接的概念。在此设定下，我们证明了存在无法被此类网络逼近的Lipschitz连续函数。这揭示了具有稠密层的神经网络的内在局限性，并表明稀疏连接是实现真正通用逼近的必要条件。