We investigate the approximation capabilities of dense neural networks. While universal approximation theorems establish that sufficiently large architectures can approximate arbitrary continuous functions if there are no restrictions on the weight values, we show that dense neural networks do not possess this universality. Our argument is based on a model compression approach, combining the weak regularity lemma with an interpretation of feedforward networks as message passing graph neural networks. We consider ReLU neural networks subject to natural constraints on weights and input and output dimensions, which model a notion of dense connectivity. Within this setting, we demonstrate the existence of Lipschitz continuous functions that cannot be approximated by such networks. This highlights intrinsic limitations of neural networks with dense layers and motivates the use of sparse connectivity as a necessary ingredient for achieving true universality.

翻译：我们研究了稠密神经网络的近似能力。尽管通用近似定理表明，若权重值无限制，足够大的架构可以逼近任意连续函数，但我们证明稠密神经网络并不具备这一通用性。我们的论证基于模型压缩方法，将弱正则引理与前馈网络作为消息传递图神经网络的解释相结合。我们考虑在权重、输入和输出维度上施加自然约束的ReLU神经网络，这类约束模拟了稠密连接的概念。在此框架下，我们证明了存在Lipschitz连续函数无法被此类网络逼近。这突显了具有稠密层的神经网络的内在局限性，并表明稀疏连接是实现真正通用性的必要成分。