The unified streaming and non-streaming speech recognition model has achieved great success due to its comprehensive capabilities. In this paper, we propose to improve the accuracy of the unified model by bridging the inherent representation gap between the streaming and non-streaming modes with a contrastive objective. Specifically, the top-layer hidden representation at the same frame of the streaming and non-streaming modes are regarded as a positive pair, encouraging the representation of the streaming mode close to its non-streaming counterpart. The multiple negative samples are randomly selected from the rest frames of the same sample under the non-streaming mode. Experimental results demonstrate that the proposed method achieves consistent improvements toward the unified model in both streaming and non-streaming modes. Our method achieves CER of 4.66% in the streaming mode and CER of 4.31% in the non-streaming mode, which sets a new state-of-the-art on the AISHELL-1 benchmark.

翻译：统一流式与非流式语音识别模型因其全面能力而取得了巨大成功。本文通过对比目标，弥合流式与非流式模式之间固有的表示差异，从而提出改进统一模型准确性的方法。具体而言，将流式与非流式模式下同一帧的顶层隐藏表示视为正样本对，促使流式模式的表示接近其非流式对应项。多个负样本从同一样本在非流式模式下的其余帧中随机选取。实验结果表明，所提方法在流式与非流式模式下均对统一模型带来了一致性改进。我们的方法在流式模式下实现了4.66%的字符错误率（CER），在非流式模式下实现了4.31%的CER，在AISHELL-1基准测试上创下了新的最优性能记录。