Trained attention layers exhibit striking and reproducible spectral structure of the weights, including low-rank collapse, bulk deformation, and isolated spectral outliers, yet the origin of these phenomena and their implications for generalization remain poorly understood. We study empirical risk minimization in a single-head tied-attention layer trained on synthetic high-dimensional sequence tasks generated from the attention-indexed model. Using tools from random matrix theory, spin-glass theory, and approximate message passing, we obtain an exact high-dimensional characterization of training and test error, interpolation and recovery thresholds, and the spectrum of the key and query matrices. Our theory predicts the full singular-value distribution of the trained query-key map, including low-rank structure and isolated spectral outliers, in qualitative agreement with observations in more realistic transformers. Finally, for targets with power-law spectra, we show that learning proceeds through sequential spectral recovery, leading to the emergence of power-law scaling laws.

翻译：训练后的注意力层展现出显著且可复现的权重谱结构，包括低秩坍缩、体谱变形与孤立谱异常值，然而这些现象的起源及其对泛化的影响仍不甚明晰。我们在由注意力索引模型生成的高维合成序列任务上，研究单头绑定注意力层的经验风险最小化问题。借助随机矩阵理论、自旋玻璃理论与近似消息传递等工具，我们获得了训练与测试误差、插值与恢复阈值以及键矩阵和查询矩阵谱的精确高维刻画。我们的理论预测了训练后查询-键映射的完整奇异值分布，包括低秩结构与孤立谱异常值，其定性特征与更现实Transformer中的观测结果一致。最后，对于具有幂律谱的目标函数，我们证明学习通过顺序谱恢复进行，从而引致幂律标度律的出现。