This paper studies the approximation capabilities of neural networks that combine layer normalization (LN) with linear layers. We prove that networks consisting of two linear layers with parallel layer normalizations (PLNs) inserted between them (referred to as PLN-Nets) achieve universal approximation, whereas architectures that use only standard LN exhibit strictly limited expressive power.We further analyze approximation rates of shallow and deep PLN-Nets under the $L^\infty$ norm as well as in Sobolev norms. Our analysis extends beyond LN to RMSNorm, and from standard MLPs to position-wise feed-forward networks, the core building blocks used in RNNs and Transformers.Finally, we provide empirical experiments to explore other possible potentials of PLN-Nets.

翻译：本文研究了结合层归一化(LN)与线性层的神经网络的逼近能力。我们证明了由两个线性层组成、并在其间插入并行层归一化(PLN)的网络(称为PLN-Net)能够实现通用逼近，而仅使用标准LN的架构则表现出严格受限的表达能力。我们进一步分析了浅层和深层PLN-Net在$L^\infty$范数及Sobolev范数下的逼近速率。我们的分析不仅适用于LN，还扩展至RMSNorm；不仅涵盖标准MLP，还包含RNN和Transformer中使用的核心构建模块——逐位置前馈网络。最后，我们通过实证实验探索了PLN-Net的其他潜在优势。