In the evolving landscape of machine learning, a pivotal challenge lies in deciphering the internal representations harnessed by neural networks and Transformers. Building on recent progress toward comprehending how networks execute distinct target functions, our study embarks on an exploration of the underlying reasons behind networks adopting specific computational strategies. We direct our focus to the complex algebraic learning task of modular addition involving $k$ inputs. Our research presents a thorough analytical characterization of the features learned by stylized one-hidden layer neural networks and one-layer Transformers in addressing this task. A cornerstone of our theoretical framework is the elucidation of how the principle of margin maximization shapes the features adopted by one-hidden layer neural networks. Let $p$ denote the modulus, $D_p$ denote the dataset of modular arithmetic with $k$ inputs and $m$ denote the network width. We demonstrate that a neuron count of $ m \geq 2^{2k-2} \cdot (p-1) $, these networks attain a maximum $ L_{2,k+1} $-margin on the dataset $ D_p $. Furthermore, we establish that each hidden-layer neuron aligns with a specific Fourier spectrum, integral to solving modular addition problems. By correlating our findings with the empirical observations of similar studies, we contribute to a deeper comprehension of the intrinsic computational mechanisms of neural networks. Furthermore, we observe similar computational mechanisms in attention matrices of one-layer Transformers. Our work stands as a significant stride in unraveling their operation complexities, particularly in the realm of complex algebraic tasks.

翻译：在机器学习的演进图景中，一个关键挑战在于破译神经网络和Transformer所利用的内部表示。基于近期在理解网络如何执行特定目标函数方面取得的进展，本研究着手探索网络采用特定计算策略背后的根本原因。我们将焦点集中于涉及 $k$ 个输入的模加法这一复杂代数学习任务。我们的研究对风格化的单隐藏层神经网络和单层Transformer在解决此任务时所学到的特征进行了全面的分析性刻画。我们理论框架的一个基石是阐明边际最大化原则如何塑造单隐藏层神经网络所采用的特征。令 $p$ 表示模数，$D_p$ 表示具有 $k$ 个输入的模运算数据集，$m$ 表示网络宽度。我们证明，当神经元数量满足 $ m \geq 2^{2k-2} \cdot (p-1) $ 时，这些网络在数据集 $ D_p $ 上达到最大 $ L_{2,k+1} $-边际。此外，我们确定每个隐藏层神经元都与特定的傅里叶谱相对应，这对于解决模加法问题至关重要。通过将我们的发现与类似研究的实证观察相关联，我们有助于更深入地理解神经网络的内在计算机制。此外，我们在单层Transformer的注意力矩阵中观察到了类似的计算机制。我们的工作是在揭示其操作复杂性，尤其是在复杂代数任务领域方面迈出的重要一步。