The underlying mechanism of neural networks in capturing precise knowledge has been the subject of consistent research efforts. In this work, we propose a theoretical approach based on Neural Tangent Kernels (NTKs) to investigate such mechanisms. Specifically, considering the infinite network width, we hypothesize the learning dynamics of target models may intuitively unravel the features they acquire from training data, deepening our insights into their internal mechanisms. We apply our approach to several fundamental models and reveal how these models leverage statistical features during gradient descent and how they are integrated into final decisions. We also discovered that the choice of activation function can affect feature extraction. For instance, the use of the \textit{ReLU} activation function could potentially introduce a bias in features, providing a plausible explanation for its replacement with alternative functions in recent pre-trained language models. Additionally, we find that while self-attention and CNN models may exhibit limitations in learning n-grams, multiplication-based models seem to excel in this area. We verify these theoretical findings through experiments and find that they can be applied to analyze language modeling tasks, which can be regarded as a special variant of classification. Our contributions offer insights into the roles and capacities of fundamental components within large language models, thereby aiding the broader understanding of these complex systems.

翻译：神经网络在捕获精确知识方面的潜在机制一直是持续研究的主题。本文基于神经正切核（NTKs）提出了一种理论方法来探究此类机制。具体而言，考虑无限网络宽度，我们假设目标模型的学习动态可能直观地揭示其从训练数据中获取的特征，从而加深对其内部机制的理解。我们将该方法应用于几个基础模型，揭示了这些模型如何在梯度下降过程中利用统计特征，以及这些特征如何被整合到最终决策中。我们还发现，激活函数的选择会影响特征提取。例如，使用\textit{ReLU}激活函数可能会在特征中引入偏差，这为近期预训练语言模型中用其他函数替代它提供了合理解释。此外，我们发现自注意力机制和CNN模型在学习n-gram时可能表现出局限性，而基于乘法的模型似乎在此方面更胜一筹。我们通过实验验证了这些理论发现，并发现它们可用于分析语言建模任务，这可视作分类任务的一种特殊变体。我们的贡献在于洞悉了大语言模型基本组件的角色与能力，从而有助于更广泛地理解这些复杂系统。