This paper introduces a novel Token-and-Duration Transducer (TDT) architecture for sequence-to-sequence tasks. TDT extends conventional RNN-Transducer architectures by jointly predicting both a token and its duration, i.e. the number of input frames covered by the emitted token. This is achieved by using a joint network with two outputs which are independently normalized to generate distributions over tokens and durations. During inference, TDT models can skip input frames guided by the predicted duration output, which makes them significantly faster than conventional Transducers which process the encoder output frame by frame. TDT models achieve both better accuracy and significantly faster inference than conventional Transducers on different sequence transduction tasks. TDT models for Speech Recognition achieve better accuracy and up to 2.82X faster inference than conventional Transducers. TDT models for Speech Translation achieve an absolute gain of over 1 BLEU on the MUST-C test compared with conventional Transducers, and its inference is 2.27X faster. In Speech Intent Classification and Slot Filling tasks, TDT models improve the intent accuracy by up to over 1% (absolute) over conventional Transducers, while running up to 1.28X faster. Our implementation of the TDT model will be open-sourced with the NeMo (https://github.com/NVIDIA/NeMo) toolkit.

翻译：本文提出了一种新颖的令牌-时长变换器（Token-and-Duration Transducer, TDT）架构，用于序列到序列任务。TDT扩展了传统的RNN-变换器架构，通过联合预测一个令牌及其时长（即该令牌所覆盖的输入帧数）。这是通过使用具有两个输出的联合网络实现的，这两个输出分别独立归一化，以生成令牌和时长上的概率分布。在推理过程中，TDT模型可以根据预测的时长输出跳过输入帧，这使得它们比逐帧处理编码器输出的传统变换器显著更快。在不同的序列转换任务中，TDT模型在准确性和推理速度方面均优于传统变换器。用于语音识别的TDT模型在准确性上优于传统变换器，并且推理速度快达2.82倍。用于语音翻译的TDT模型在MUST-C测试集上相较于传统变换器获得了超过1个BLEU的绝对增益，且推理速度快2.27倍。在语音意图分类和槽位填充任务中，TDT模型相较于传统变换器将意图准确率绝对提升了高达1%以上，同时运行速度快达1.28倍。我们的TDT模型实现将随NeMo工具包（https://github.com/NVIDIA/NeMo）开源。