The underlying mechanism of neural networks in capturing precise knowledge has been the subject of consistent research efforts. In this work, we propose a theoretical approach based on Neural Tangent Kernels (NTKs) to investigate such mechanisms. Specifically, considering the infinite network width, we hypothesize the learning dynamics of target models may intuitively unravel the features they acquire from training data, deepening our insights into their internal mechanisms. We apply our approach to several fundamental models and reveal how these models leverage statistical features during gradient descent and how they are integrated into final decisions. We also discovered that the choice of activation function can affect feature extraction. For instance, the use of the \textit{ReLU} activation function could potentially introduce a bias in features, providing a plausible explanation for its replacement with alternative functions in recent pre-trained language models. Additionally, we find that while self-attention and CNN models may exhibit limitations in learning n-grams, multiplication-based models seem to excel in this area. We verify these theoretical findings through experiments and find that they can be applied to analyze language modeling tasks, which can be regarded as a special variant of classification. Our contributions offer insights into the roles and capacities of fundamental components within large language models, thereby aiding the broader understanding of these complex systems.

翻译：神经网络捕获精确知识的潜在机制一直是持续研究的课题。本文提出一种基于神经正切核（Neural Tangent Kernels, NTKs）的理论方法来探究此类机制。具体而言，考虑网络宽度趋于无穷的情况，我们假设目标模型的学习动态可能直观地揭示其从训练数据中获取的特征，从而深化对其内部机制的理解。我们将该方法应用于若干基础模型，揭示了这些模型在梯度下降过程中如何利用统计特征，以及这些特征如何整合到最终决策中。我们还发现激活函数的选择会影响特征提取。例如，使用\textit{ReLU}激活函数可能引入特征偏差，这为近期预训练语言模型用其他函数替代ReLU提供了合理解释。此外，我们发现自注意力模型和CNN在学习n-gram时可能表现有限，而基于乘法的模型在此方面似乎更具优势。我们通过实验验证了这些理论发现，并证明其可用于分析可视为分类特殊变体的语言建模任务。本文的贡献在于揭示了大语言模型基本组件的作用与能力，从而有助于更广泛地理解这些复杂系统。