Modern Hopfield networks (MHNs) have emerged as powerful tools in deep learning, capable of replacing components such as pooling layers, LSTMs, and attention mechanisms. Recent advancements have enhanced their storage capacity, retrieval speed, and error rates. However, the fundamental limits of their computational expressiveness remain unexplored. Understanding the expressive power of MHNs is crucial for optimizing their integration into deep learning architectures. In this work, we establish rigorous theoretical bounds on the computational capabilities of MHNs using circuit complexity theory. Our key contribution is that we show that MHNs are $\mathsf{DLOGTIME}$-uniform $\mathsf{TC}^0$. Hence, unless $\mathsf{TC}^0 = \mathsf{NC}^1$, a $\mathrm{poly}(n)$-precision modern Hopfield networks with a constant number of layers and $O(n)$ hidden dimension cannot solve $\mathsf{NC}^1$-hard problems such as the undirected graph connectivity problem and the tree isomorphism problem. We also extended our results to Kernelized Hopfield Networks. These results demonstrate the limitation in the expressive power of the modern Hopfield networks. Moreover, Our theoretical analysis provides insights to guide the development of new Hopfield-based architectures.

翻译：现代Hopfield网络（MHNs）已成为深度学习中的强大工具，能够替代池化层、LSTM和注意力机制等组件。最新进展提升了其存储容量、检索速度和错误率。然而，其计算表达能力的基本界限仍未得到探索。理解MHNs的表达能力对于优化其在深度学习架构中的集成至关重要。本研究利用电路复杂性理论，为MHNs的计算能力建立了严格的理论界限。我们的核心贡献在于证明MHNs属于$\mathsf{DLOGTIME}$-均匀$\mathsf{TC}^0$类。因此，除非$\mathsf{TC}^0 = \mathsf{NC}^1$，否则具有常数层数和$O(n)$隐藏维度、精度为$\mathrm{poly}(n)$的现代Hopfield网络无法解决$\mathsf{NC}^1$困难问题，例如无向图连通性问题和树同构问题。我们还将结果扩展至核化Hopfield网络。这些结论揭示了现代Hopfield网络在表达能力上的局限性。此外，我们的理论分析为开发新型基于Hopfield的架构提供了指导性见解。