Attention layers, as commonly used in transformers, form the backbone of modern deep learning, yet there is no mathematical description of their benefits and deficiencies as compared with other architectures. In this work we establish both positive and negative results on the representation power of attention layers, with a focus on intrinsic complexity parameters such as width, depth, and embedding dimension. On the positive side, we present a sparse averaging task, where recurrent networks and feedforward networks all have complexity scaling polynomially in the input size, whereas transformers scale merely logarithmically in the input size; furthermore, we use the same construction to show the necessity and role of a large embedding dimension in a transformer. On the negative side, we present a triple detection task, where attention layers in turn have complexity scaling linearly in the input size; as this scenario seems rare in practice, we also present natural variants that can be efficiently solved by attention layers. The proof techniques emphasize the value of communication complexity in the analysis of transformers and related models, and the role of sparse averaging as a prototypical attention task, which even finds use in the analysis of triple detection.

翻译：注意力层作为Transformer的核心组件，支撑着现代深度学习的发展，然而目前尚缺乏对其相较于其他架构之优劣的数学描述。本研究聚焦宽度、深度及嵌入维度等内在复杂度参数，系统建立注意力层表征能力的正反两方面结论。在正向结果方面，我们提出稀疏平均任务：递归网络与前馈网络的复杂度随输入规模呈多项式增长，而Transformer仅呈对数增长；进而通过相同构造证明大嵌入维度在Transformer中的必要性与作用。在反向结果方面，我们提出三重检测任务，其中注意力层的复杂度随输入规模呈线性增长；鉴于该场景在实践中较为罕见，我们同时提出注意力层可高效求解的自然变体。证明技术凸显了通信复杂度在Transformer及相关模型分析中的价值，并揭示稀疏平均作为典型注意力任务的核心作用——该任务甚至可应用于三重检测分析。