In recent years, self-attention has become the dominant paradigm for sequence modeling in a variety of domains. However, in domains with very long sequence lengths the $\mathcal{O}(T^2)$ memory and $\mathcal{O}(T^2 H)$ compute costs can make using transformers infeasible. Motivated by problems in malware detection, where sequence lengths of $T \geq 100,000$ are a roadblock to deep learning, we re-cast self-attention using the neuro-symbolic approach of Holographic Reduced Representations (HRR). In doing so we perform the same high-level strategy of the standard self-attention: a set of queries matching against a set of keys, and returning a weighted response of the values for each key. Implemented as a ``Hrrformer'' we obtain several benefits including $\mathcal{O}(T H \log H)$ time complexity, $\mathcal{O}(T H)$ space complexity, and convergence in $10\times$ fewer epochs. Nevertheless, the Hrrformer achieves near state-of-the-art accuracy on LRA benchmarks and we are able to learn with just a single layer. Combined, these benefits make our Hrrformer the first viable Transformer for such long malware classification sequences and up to $280\times$ faster to train on the Long Range Arena benchmark. Code is available at \url{https://github.com/NeuromorphicComputationResearchProgram/Hrrformer}

翻译：近年来，自注意力机制已成为多个领域中序列建模的主导范式。然而，在序列长度极长的场景下，$\mathcal{O}(T^2)$的空间复杂度与$\mathcal{O}(T^2 H)$的计算复杂度使得Transformer难以实际应用。受恶意软件检测问题的驱动（其中$T \geq 100,000$的序列长度构成深度学习的障碍），我们采用神经符号方法——全息简化表征（HRR），对自注意力机制进行重构。通过该方法，我们实现了与标准自注意力相同的高层策略：一组查询与一组键进行匹配，并返回每个键对应值的加权响应。所实现的"Hrrformer"具有多项优势，包括$\mathcal{O}(T H \log H)$的时间复杂度、$\mathcal{O}(T H)$的空间复杂度，以及收敛速度提升10倍的训练效率。尽管如此，Hrrformer在LRA基准测试中仍能达到接近最优的准确率，且仅需单层即可完成学习。综合这些优势，我们的Hrrformer成为首个能够有效处理超长恶意软件分类序列的Transformer变体，在Long Range Arena基准测试上的训练速度最高提升280倍。代码已开源：\url{https://github.com/NeuromorphicComputationResearchProgram/Hrrformer}