Attention layers, as commonly used in transformers, form the backbone of modern deep learning, yet there is no mathematical description of their benefits and deficiencies as compared with other architectures. In this work we establish both positive and negative results on the representation power of attention layers, with a focus on intrinsic complexity parameters such as width, depth, and embedding dimension. On the positive side, we present a sparse averaging task, where recurrent networks and feedforward networks all have complexity scaling polynomially in the input size, whereas transformers scale merely logarithmically in the input size; furthermore, we use the same construction to show the necessity and role of a large embedding dimension in a transformer. On the negative side, we present a triple detection task, where attention layers in turn have complexity scaling linearly in the input size; as this scenario seems rare in practice, we also present natural variants that can be efficiently solved by attention layers. The proof techniques emphasize the value of communication complexity in the analysis of transformers and related models, and the role of sparse averaging as a prototypical attention task, which even finds use in the analysis of triple detection.

翻译：注意力层作为现代深度学习的基础组件，广泛应用于Transformer架构中，但相较于其他架构其优势与缺陷尚缺乏数学描述。本研究针对注意力层的表征能力建立了正反两方面结论，重点关注宽度、深度及嵌入维度等内在复杂度参数。在正面结果方面，我们提出稀疏平均任务：在该任务中循环网络与前馈网络的复杂度随输入规模呈多项式增长，而Transformer仅呈对数增长；同时利用相同构造证明了大嵌入维度在Transformer中的必要性及其作用。在负面结果方面，我们提出三重检测任务：在该任务中注意力层的复杂度呈线性增长；鉴于此类场景在实践中较为罕见，我们同时给出了注意力层可高效求解的若干自然变体。本文证明技术凸显了通信复杂度在Transformer及同类模型分析中的价值，并表明稀疏平均作为原型注意力任务的作用，该任务甚至可用于三重检测分析。