Transformer models have achieved remarkable results in a wide range of applications. However, their scalability is hampered by the quadratic time and memory complexity of the self-attention mechanism concerning the sequence length. This limitation poses a substantial obstacle when dealing with long documents or high-resolution images. In this work, we study the self-attention mechanism by analyzing the distribution of the attention matrix and its concentration ability. Furthermore, we propose instruments to measure these quantities and introduce a novel self-attention mechanism, Linear Log-Normal Attention, designed to emulate the distribution and concentration behavior of the original self-attention. Our experimental results on popular natural language benchmarks reveal that our proposed Linear Log-Normal Attention outperforms other linearized attention alternatives, offering a promising avenue for enhancing the scalability of transformer models.

翻译：Transformer模型在广泛应用中取得了显著成果。然而，其自注意力机制因序列长度导致的时间与内存复杂度呈二次增长，严重制约了模型的可扩展性，尤其在处理长文档或高分辨率图像时成为重大障碍。本研究通过分析注意力矩阵的分布特性及其集中能力，深入探究自注意力机制。我们进一步提出了量化这些特性的指标，并设计了一种新型自注意力机制——线性对数正态注意力（Linear Log-Normal Attention），旨在模拟原始自注意力的分布与集中行为。在主流自然语言基准上的实验结果表明，我们提出的线性对数正态注意力优于其他线性化注意力方案，为提升Transformer模型的可扩展性提供了有前景的途径。