We present a nonparametric construction for deep learning compatible modern Hopfield models and utilize this framework to debut an efficient variant. Our key contribution stems from interpreting the memory storage and retrieval processes in modern Hopfield models as a nonparametric regression problem subject to a set of query-memory pairs. Crucially, our framework not only recovers the known results from the original dense modern Hopfield model but also fills the void in the literature regarding efficient modern Hopfield models, by introducing \textit{sparse-structured} modern Hopfield models with sub-quadratic complexity. We establish that this sparse model inherits the appealing theoretical properties of its dense analogue -- connection with transformer attention, fixed point convergence and exponential memory capacity -- even without knowing details of the Hopfield energy function. Additionally, we showcase the versatility of our framework by constructing a family of modern Hopfield models as extensions, including linear, random masked, top-$K$ and positive random feature modern Hopfield models. Empirically, we validate the efficacy of our framework in both synthetic and realistic settings.

翻译：我们提出了一种与深度学习兼容的现代Hopfield模型的非参数化构建方法，并利用该框架首次推出一种高效变体。我们的核心贡献在于将现代Hopfield模型中的记忆存储与检索过程解释为受查询-记忆对约束的非参数回归问题。关键的是，该框架不仅恢复了原始密集现代Hopfield模型的已知结果，还通过引入具有亚二次复杂度的\emph{稀疏结构化}现代Hopfield模型，填补了高效现代Hopfield模型领域的空白。我们证明，即使不了解Hopfield能量函数的细节，该稀疏模型仍然继承了其密集对应物的优良理论性质——与Transformer注意力机制的联系、不动点收敛性以及指数级记忆容量。此外，我们通过构建一系列现代Hopfield模型扩展（包括线性、随机掩码、top-$K$和正随机特征现代Hopfield模型），展示了该框架的通用性。在实验上，我们在合成和真实场景中验证了该框架的有效性。