Recently, neural networks utilizing periodic activation functions have been proven to demonstrate superior performance in vision tasks compared to traditional ReLU-activated networks. However, there is still a limited understanding of the underlying reasons for this improved performance. In this paper, we aim to address this gap by providing a theoretical understanding of periodically activated networks through an analysis of their Neural Tangent Kernel (NTK). We derive bounds on the minimum eigenvalue of their NTK in the finite width setting, using a fairly general network architecture which requires only one wide layer that grows at least linearly with the number of data samples. Our findings indicate that periodically activated networks are \textit{notably more well-behaved}, from the NTK perspective, than ReLU activated networks. Additionally, we give an application to the memorization capacity of such networks and verify our theoretical predictions empirically. Our study offers a deeper understanding of the properties of periodically activated neural networks and their potential in the field of deep learning.

翻译：近期研究表明，采用周期性激活函数的神经网络在视觉任务中展现出优于传统ReLU激活网络的性能。然而，这种性能提升的内在机理仍缺乏深入理解。本文旨在通过分析周期性激活网络的神经正切核（NTK），为其提供理论解释。我们推导了在有限宽度设置下NTK最小特征值的边界，所采用的网络结构具有相当普适性，仅需单个宽度随数据样本数量线性增长的宽层即可实现。研究结果表明，从NTK视角来看，周期性激活网络比ReLU激活网络表现出显著更优的稳定性。此外，我们将该理论应用于此类网络的记忆容量分析，并通过实验验证了理论预测。本研究为周期性激活神经网络的特性及其在深度学习领域的应用潜力提供了更深入的理论认知。