We introduce an Outlier-Efficient Modern Hopfield Model (termed $\mathtt{OutEffHop}$) and use it to address the outlier-induced challenge of quantizing gigantic transformer-based models. Our main contribution is a novel associative memory model facilitating \textit{outlier-efficient} associative memory retrievals. Interestingly, this memory model manifests a model-based interpretation of an outlier-efficient attention mechanism ($\text{Softmax}_1$): it is an approximation of the memory retrieval process of $\mathtt{OutEffHop}$. Methodologically, this allows us to debut novel outlier-efficient Hopfield layers a powerful attention alternative with superior post-quantization performance. Theoretically, the Outlier-Efficient Modern Hopfield Model retains and improves the desirable properties of the standard modern Hopfield models, including fixed point convergence and exponential storage capacity. Empirically, we demonstrate the proposed model's efficacy across large-scale transformer-based and Hopfield-based models (including BERT, OPT, ViT and STanHop-Net), benchmarking against state-of-the-art methods including $\mathtt{Clipped\_Softmax}$ and $\mathtt{Gated\_Attention}$. Notably, $\mathtt{OutEffHop}$ achieves on average $\sim$22+\% reductions in both average kurtosis and maximum infinity norm of model outputs accross 4 models.

翻译：我们提出了一种离群高效现代Hopfield模型（称为$\mathtt{OutEffHop}$），并将其用于解决量化大型Transformer模型时由离群值引发的挑战。我们的主要贡献在于提出了一种新型联想记忆模型，能够实现\textit{离群高效}的联想记忆检索。有趣的是，该记忆模型为离群高效注意力机制（$\text{Softmax}_1$）提供了一种基于模型的解释：它是$\mathtt{OutEffHop}$记忆检索过程的近似。在方法论上，这使我们得以首次提出离群高效Hopfield层——一种具有优越后量化性能的强大注意力替代方案。理论上，离群高效现代Hopfield模型保留并改进了标准现代Hopfield模型的理想特性，包括不动点收敛性和指数级存储容量。在实证层面，我们展示了所提模型在基于Transformer和Hopfield的大规模模型（包括BERT、OPT、ViT和STanHop-Net）中的有效性，并与$\mathtt{Clipped\_Softmax}$和$\mathtt{Gated\_Attention}$等最新方法进行了基准对比。值得注意的是，$\mathtt{OutEffHop}$在4个模型上的平均峰度和最大无穷范数平均降低了约22%以上。