Conformer-based attention models have become the de facto backbone model for Automatic Speech Recognition tasks. A blank symbol is usually introduced to align the input and output sequences for CTC or RNN-T models. Unfortunately, the long input length overloads computational budget and memory consumption quadratically by attention mechanism. In this work, we propose a "Skip-and-Recover" Conformer architecture, named Skipformer, to squeeze sequence input length dynamically and inhomogeneously. Skipformer uses an intermediate CTC output as criteria to split frames into three groups: crucial, skipping and ignoring. The crucial group feeds into next conformer blocks and its output joint with skipping group by original temporal order as the final encoder output. Experiments show that our model reduces the input sequence length by 31 times on Aishell-1 and 22 times on Librispeech corpus. Meanwhile, the model can achieve better recognition accuracy and faster inference speed than recent baseline models. Our code is open-sourced and available online.

翻译：基于Conformer的注意力模型已成为自动语音识别任务的事实骨干模型。CTC或RNN-T模型通常引入空白符号以对齐输入与输出序列。然而，过长的输入长度会因注意力机制而二次方地增加计算负担与内存消耗。本研究提出一种名为Skipformer的“跳跃与恢复”Conformer架构，可动态且非均匀地压缩序列输入长度。Skipformer以中间CTC输出为准则，将帧划分为三类：关键帧、跳跃帧与忽略帧。关键帧被送入后续Conformer模块，其输出与跳跃帧按原始时序合并，作为最终编码器输出。实验表明，本模型在Aishell-1数据集上可将输入序列长度压缩31倍，在Librispeech数据集上压缩22倍。同时，该模型相较于近期基线模型，能实现更优的识别准确率与更快的推理速度。相关代码已开源并在线发布。