Transformer models have achieved remarkable results in a wide range of applications. However, their scalability is hampered by the quadratic time and memory complexity of the self-attention mechanism concerning the sequence length. This limitation poses a substantial obstacle when dealing with long documents or high-resolution images. In this work, we study the self-attention mechanism by analyzing the distribution of the attention matrix and its concentration ability. Furthermore, we propose instruments to measure these quantities and introduce a novel self-attention mechanism, Linear Log-Normal Attention, designed to emulate the distribution and concentration behavior of the original self-attention. Our experimental results on popular natural language benchmarks reveal that our proposed Linear Log-Normal Attention outperforms other linearized attention alternatives, offering a promising avenue for enhancing the scalability of transformer models. Our code is available in supplementary materials.

翻译：Transformer模型在广泛应用中取得了显著成果。然而，其自注意力机制因序列长度呈二次方的时间和内存复杂度，严重限制了模型的可扩展性。这一局限在处理长文档或高分辨率图像时尤为突出。本文通过分析注意力矩阵的分布及集中能力，对自注意力机制展开了深入研究。我们进一步提出了量化这些特征的指标，并引入了一种新型自注意力机制——线性对数正态注意力，旨在模拟原始自注意力的分布与集中行为。在主流自然语言基准上的实验结果表明，我们提出的线性对数正态注意力优于其他线性化注意力替代方案，为提升Transformer模型的可扩展性开辟了有前景的途径。相关代码见补充材料。