Recent generalizations of the Hopfield model of associative memories are able to store a number $P$ of random patterns that grows exponentially with the number $N$ of neurons, $P=\exp(\alpha N)$. Besides the huge storage capacity, another interesting feature of these networks is their connection to the attention mechanism which is part of the Transformer architectures widely applied in deep learning. In this work, we consider a generic family of pattern ensembles, and thanks to the statistical mechanics analysis of an auxiliary Random Energy Model, we are able to provide exact asymptotic thresholds for the retrieval of a typical pattern, $\alpha_1$, and lower bounds for the maximum of the load $\alpha$ for which all patterns can be retrieved, $\alpha_c$. Additionally, we characterize the size of the basins of attractions. We discuss in detail the cases of Gaussian and spherical patterns, and show that they display rich and qualitatively different phase diagrams.

翻译：近期对Hopfield联想记忆模型的推广能够存储数量随神经元数量N呈指数增长的随机模式P，即$P=\exp(\alpha N)$。除了巨大的存储容量外，这类网络的另一个有趣特征是其与注意力机制的联系——该机制是深度学习中广泛应用的Transformer架构的组成部分。在本工作中，我们考虑了一类通用的模式系综，借助辅助随机能量模型的统计力学分析，成功推导出典型模式检索的精确渐近阈值$\alpha_1$，以及所有模式均可检索时负载$\alpha$最大值$\alpha_c$的下界。此外，我们还刻画了吸引域的大小。我们详细讨论了高斯模式和球面模式的情形，并表明它们呈现出丰富且定性不同的相图。