The attention mechanism is an important reason for the success of transformers. It relies on computing pairwise relations between tokens. To reduce the high computational cost of standard quadratic attention, linear attention has been proposed as an efficient approximation. It employs kernel functions that are applied independently to the inputs before the pairwise similarities are calculated. That allows for an efficient computational procedure which, however, amounts to a low-degree rational function approximating softmax. We propose Hadamard Linear Attention (HLA). Unlike previous works on linear attention, the nonlinearity in HLA is not applied separately to queries and keys, but, analogously to standard softmax attention, after the pairwise similarities have been computed. It will be shown that the proposed nonlinearity amounts to a higher-degree rational function to approximate softmax. An efficient computational scheme for the proposed method is derived that is similar to that of standard linear attention. In contrast to other approaches, no time-consuming tensor reshaping is necessary to apply the proposed algorithm. The effectiveness of the approach is demonstrated by applying it to a large diffusion transformer model for video generation, an application that involves very large amounts of tokens.

翻译：注意力机制是Transformer模型取得成功的重要原因。它依赖于计算标记之间的成对关系。为了降低标准二次注意力的高计算成本，线性注意力作为一种高效近似方法被提出。该方法采用核函数，在计算成对相似度之前独立应用于输入。这虽然实现了高效的计算流程，但仅相当于用低阶有理函数逼近softmax。本文提出哈达玛线性注意力（HLA）。与以往线性注意力研究不同，HLA中的非线性并非分别应用于查询和键，而是类比标准softmax注意力，在计算完成对相似度后实施。研究表明，所提出的非线性相当于用更高阶的有理函数逼近softmax。本文推导出与标准线性注意力相似的高效计算方案。与其他方法相比，该算法无需耗时的张量重塑操作。通过将HLA应用于视频生成的大型扩散Transformer模型（该应用涉及海量标记处理），验证了该方法的有效性。