The recent discovery of a connection between Transformers and Modern Hopfield Networks (MHNs) has reignited the study of neural networks from a physical energy-based perspective. This paper focuses on the pivotal effect of the inverse temperature hyperparameter $\beta$ on the distribution of energy minima of the MHN. To achieve this, the distribution of energy minima is tracked in a simplified MHN in which equidistant normalised patterns are stored. This network demonstrates a phase transition at a critical temperature $\beta_{\text{c}}$, from a single global attractor towards highly pattern specific minima as $\beta$ is increased. Importantly, the dynamics are not solely governed by the hyperparameter $\beta$ but are instead determined by an effective inverse temperature $\beta_{\text{eff}}$ which also depends on the distribution and size of the stored patterns. Recognizing the role of hyperparameters in the MHN could, in the future, aid researchers in the domain of Transformers to optimise their initial choices, potentially reducing the necessity for time and energy expensive hyperparameter fine-tuning.

翻译：近期，Transformer与现代霍普菲尔德网络之间关联的发现，从基于物理能量的角度重新点燃了对神经网络的研究。本文聚焦于逆温度超参数$\beta$对现代霍普菲尔德网络能量最小值分布的关键影响。为此，我们追踪存储等距归一化模式的简化现代霍普菲尔德网络中能量最小值的分布。该网络在临界温度$\beta_{\text{c}}$处展现相变，随着$\beta$增大，网络从单一全局吸引子过渡到高度模式特异的最小值。重要的是，动力学并非仅由超参数$\beta$单独控制，而是由有效逆温度$\beta_{\text{eff}}$决定，该参数还依赖于存储模式的分布与规模。认识到现代霍普菲尔德网络中超参数的作用，未来或可帮助Transformer领域的研究者优化初始选择，从而可能减少耗时且耗能的大量超参数调优需求。