Transformer-based models excel in speech recognition. Existing efforts to optimize Transformer inference, typically for long-context applications, center on simplifying attention score calculations. However, streaming speech recognition models usually process a limited number of tokens each time, making attention score calculation less of a bottleneck. Instead, the bottleneck lies in the linear projection layers of multi-head attention and feedforward networks, constituting a substantial portion of the model size and contributing significantly to computation, memory, and power usage. To address this bottleneck, we propose folding attention, a technique targeting these linear layers, significantly reducing model size and improving memory and power efficiency. Experiments on on-device Transformer-based streaming speech recognition models show that folding attention reduces model size (and corresponding memory consumption) by up to 24% and power consumption by up to 23%, all without compromising model accuracy or computation overhead.

翻译：基于Transformer的模型在语音识别中表现出色。现有针对Transformer推理的优化工作（通常面向长上下文应用）主要集中在简化注意力分数计算上。然而，流式语音识别模型每次仅处理有限数量的令牌，这使得注意力分数计算并非瓶颈所在。真正的瓶颈在于多头注意力的线性投影层和前馈网络，它们占据了模型规模的绝大部分，并在计算、内存和功耗方面贡献显著。为解决这一瓶颈，我们提出折叠注意力技术，该技术针对这些线性层，显著减小模型规模并提升内存与功耗效率。在基于Transformer的设备端流式语音识别模型上的实验表明，折叠注意力可降低高达24%的模型规模（及相应的内存消耗）和高达23%的功耗，且不损害模型精度或计算开销。