Learning arguably involves the discovery and memorization of abstract rules. The aim of this paper is to study associative memory mechanisms. Our model is based on high-dimensional matrices consisting of outer products of embeddings, which relates to the inner layers of transformer language models. We derive precise scaling laws with respect to sample size and parameter size, and discuss the statistical efficiency of different estimators, including optimization-based algorithms. We provide extensive numerical experiments to validate and interpret theoretical results, including fine-grained visualizations of the stored memory associations.

翻译：学习无疑涉及抽象规则的发现与记忆。本文旨在研究联想记忆机制。我们的模型基于由嵌入向量的外积构成的高维矩阵，这与Transformer语言模型的内部层相关。我们推导出关于样本规模和参数规模的精确缩放定律，并讨论了不同估计量（包括基于优化的算法）的统计效率。我们提供了大量数值实验来验证和解释理论结果，包括存储记忆关联的细粒度可视化。