Modern speech recognition systems rely on self-attention. Unfortunately, token mixing with self-attention takes quadratic time in the length of the speech utterance, slowing down inference as well as training and increasing memory consumption. Cheaper alternatives to self-attention for ASR have been developed, but fail to consistently reach the same level of accuracy. In practice, however, the self-attention weights of trained speech recognizers take the form of a global average over time. This paper, therefore, proposes a linear-time alternative to self-attention for speech recognition. It summarises a whole utterance with the mean over vectors for all time steps. This single summary is then combined with time-specific information. We call this method ``Summary Mixing''. Introducing Summary Mixing in state-of-the-art ASR models makes it feasible to preserve or exceed previous speech recognition performance while lowering the training and inference times by up to 27% and reducing the memory budget by a factor of two.

翻译：现代语音识别系统依赖于自注意力机制。然而，基于自注意力的词元混合在语音话语长度上具有二次时间复杂度，这不仅降低了推理与训练速度，还增加了内存消耗。虽已开发出用于自动语音识别的低成本自注意力替代方案，但这些方案难以持续达到同等精度水平。实际上，训练完成的语音识别器中的自注意力权重呈现为时间维度上的全局平均形式。为此，本文提出一种用于语音识别的线性时间复杂度自注意力替代方法：通过对所有时间步的向量取均值来概括整个话语，再将这一单一概括性表示与时间特定信息相结合。我们将此方法命名为"Summary Mixing"（概括混合）。在最新语音识别模型中引入Summary Mixing，可在保持或超越原有语音识别性能的同时，将训练与推理时间降低高达27%，并将内存消耗减半。