We explore unifying a neural segmenter with two-pass cascaded encoder ASR into a single model. A key challenge is allowing the segmenter (which runs in real-time, synchronously with the decoder) to finalize the 2nd pass (which runs 900 ms behind real-time) without introducing user-perceived latency or deletion errors during inference. We propose a design where the neural segmenter is integrated with the causal 1st pass decoder to emit a end-of-segment (EOS) signal in real-time. The EOS signal is then used to finalize the non-causal 2nd pass. We experiment with different ways to finalize the 2nd pass, and find that a novel dummy frame injection strategy allows for simultaneous high quality 2nd pass results and low finalization latency. On a real-world long-form captioning task (YouTube), we achieve 2.4% relative WER and 140 ms EOS latency gains over a baseline VAD-based segmenter with the same cascaded encoder.

翻译：我们探索将神经分段器与两遍级联编码器ASR统一到单一模型中。一个关键挑战是允许分段器（实时与解码器同步运行）在不引入用户感知延迟或推理期间删除错误的情况下，完成第二遍处理（比实时落后900毫秒）。我们提出一种设计，其中神经分段器与因果第一遍解码器集成，以实时发出分段结束（EOS）信号。随后利用该EOS信号完成非因果第二遍处理。我们尝试了多种完成第二遍处理的方式，发现一种新颖的虚拟帧注入策略能够同时获得高质量的第二遍结果与低完成延迟。在实际长格式字幕生成任务（YouTube）中，与基于VAD的基线分段器结合相同级联编码器的方案相比，我们的方法实现了2.4%的相对词错误率改善和140毫秒的EOS延迟增益。