The computational intensity of Large Language Models (LLMs) is a critical bottleneck, primarily due to the $O(n^2)$ complexity of the attention mechanism in transformer architectures. Addressing this, sparse attention emerges as a key innovation, aiming to reduce computational load while maintaining model performance. This study presents a rigorous theoretical analysis of the sparsity in attention scores within LLMs, particularly under the framework of Gaussian inputs. By establishing a set of foundational assumptions and employing a methodical theoretical approach, we unravel the intrinsic characteristics of attention score sparsity and its implications on computational efficiency. Our main contribution lies in providing a detailed theoretical examination of how sparsity manifests in attention mechanisms, offering insights into the potential trade-offs between computational savings and model effectiveness. This work not only advances our understanding of sparse attention but also provides a scaffold for future research in optimizing the computational frameworks of LLMs, paving the way for more scalable and efficient AI systems.

翻译：大语言模型的算力瓶颈主要源于Transformer架构中注意力机制的$O(n^2)$复杂度，稀疏注意力作为关键创新应运而生，旨在降低计算负载的同时保持模型性能。本研究对大规模语言模型中注意力得分的稀疏性进行了严格的理论分析，尤其在服从高斯分布的输入框架下。通过建立基础假设并采用系统性理论方法，我们揭示了注意力得分稀疏性的内在特征及其对计算效率的影响。核心贡献在于从理论层面详细阐释了稀疏性在注意力机制中的表现形态，阐明了计算效率与模型效能之间的潜在权衡关系。该工作不仅深化了对稀疏注意力的理解，更为优化大语言模型计算框架提供了理论支撑，为构建可扩展的高效人工智能系统奠定了方法论基础。