Modern speech processing systems rely on self-attention. Unfortunately, token mixing with self-attention takes quadratic time in the length of the speech utterance, slowing down inference as well as training and increasing memory consumption. Cheaper alternatives to self-attention for ASR have been developed, but they fail to consistently reach the same level of accuracy. This paper, therefore, proposes a novel linear-time alternative to self-attention. It summarises an utterance with the mean over vectors for all time steps. This single summary is then combined with time-specific information. We call this method "SummaryMixing". Introducing SummaryMixing in state-of-the-art ASR models makes it feasible to preserve or exceed previous speech recognition performance while lowering the training and inference times by up to 28$\%$ and reducing the memory budget by a factor of two. The benefits of SummaryMixing can also be generalized to other speech-processing tasks, such as speech understanding.

翻译：现代语音处理系统依赖于自注意力机制。然而，使用自注意力进行令牌混合会占用语音时长平方级的时间，从而降低推理与训练速度，并增加内存消耗。针对自动语音识别（ASR）任务，已有更低成本的次优自注意力替代方案被提出，但这些方法未能稳定达到同等准确率。为此，本文提出一种新颖的线性时间复杂度自注意力替代方案。该方法通过计算所有时间步向量均值来概括语音片段，再将这一全局摘要与时间特定信息结合。我们将此方法命名为“SummaryMixing”。将SummaryMixing引入最先进的ASR模型后，不仅能保持甚至超越原有的语音识别性能，还可将训练与推理时间缩短高达28%，并将内存消耗减半。此外，SummaryMixing的优势可泛化至其他语音处理任务，例如语音理解。