Transformer models have achieved remarkable results in a wide range of applications. However, their scalability is hampered by the quadratic time and memory complexity of the self-attention mechanism concerning the sequence length. This limitation poses a substantial obstacle when dealing with long documents or high-resolution images. In this work, we study the self-attention mechanism by analyzing the distribution of the attention matrix and its concentration ability. Furthermore, we propose instruments to measure these quantities and introduce a novel self-attention mechanism, Linear Log-Normal Attention, designed to emulate the distribution and concentration behavior of the original self-attention. Our experimental results on popular natural language benchmarks reveal that our proposed Linear Log-Normal Attention outperforms other linearized attention alternatives, offering a promising avenue for enhancing the scalability of transformer models. Our code is available in supplementary materials.

翻译：Transformer模型在广泛的应用领域中取得了显著成果。然而，其自注意力机制在序列长度维度上的二次时间与空间复杂度严重制约了模型的可扩展性，这一局限在处理长文档或高分辨率图像时尤为突出。本研究通过分析注意力矩阵的分布特性及其集中能力来探究自注意力机制。我们提出了相应的量化评估工具，并创新性地设计了线性对数正态注意力机制（Linear Log-Normal Attention），旨在模拟原始自注意力的分布特性与集中行为。在主流自然语言基准测试上的实验结果表明，我们提出的线性对数正态注意力机制优于其他线性化注意力方案，为提升Transformer模型的可扩展性开辟了新路径。相关代码见补充材料。