Transformer models have achieved remarkable results in a wide range of applications. However, their scalability is hampered by the quadratic time and memory complexity of the self-attention mechanism concerning the sequence length. This limitation poses a substantial obstacle when dealing with long documents or high-resolution images. In this work, we study the self-attention mechanism by analyzing the distribution of the attention matrix and its concentration ability. Furthermore, we propose instruments to measure these quantities and introduce a novel self-attention mechanism, Linear Log-Normal Attention, designed to emulate the distribution and concentration behavior of the original self-attention. Our experimental results on popular natural language benchmarks reveal that our proposed Linear Log-Normal Attention outperforms other linearized attention alternatives, offering a promising avenue for enhancing the scalability of transformer models.

翻译：Transformer模型在各类应用中取得了显著成果。然而，其可扩展性受到自注意力机制在序列长度上的二次时间与内存复杂度的制约。这一局限性在处理长文档或高分辨率图像时构成了重大障碍。本研究通过分析注意力矩阵的分布及其集中能力，深入探究了自注意力机制。进一步地，我们提出了度量这些特性的工具，并引入了一种新颖的自注意力机制——线性对数正态注意力，旨在模拟原始自注意力的分布与集中行为。我们在主流自然语言基准上的实验结果表明，我们提出的线性对数正态注意力优于其他线性化注意力替代方案，为提升Transformer模型的可扩展性提供了一条前景广阔的途径。