Simultaneous sequence generation is a pivotal task for real-time scenarios, such as streaming speech recognition, simultaneous machine translation and simultaneous speech translation, where the target sequence is generated while receiving the source sequence. The crux of achieving high-quality generation with low latency lies in identifying the optimal moments for generating, accomplished by learning a mapping between the source and target sequences. However, existing methods often rely on task-specific heuristics for different sequence types, limiting the model's capacity to adaptively learn the source-target mapping and hindering the exploration of multi-task learning for various simultaneous tasks. In this paper, we propose a unified segment-to-segment framework (Seg2Seg) for simultaneous sequence generation, which learns the mapping in an adaptive and unified manner. During the process of simultaneous generation, the model alternates between waiting for a source segment and generating a target segment, making the segment serve as the natural bridge between the source and target. To accomplish this, Seg2Seg introduces a latent segment as the pivot between source to target and explores all potential source-target mappings via the proposed expectation training, thereby learning the optimal moments for generating. Experiments on multiple simultaneous generation tasks demonstrate that Seg2Seg achieves state-of-the-art performance and exhibits better generality across various tasks.

翻译：同步序列生成是实时场景中的关键任务，例如流式语音识别、同步机器翻译和同步语音翻译，这些场景中目标序列在接收源序列的同时生成。在低延迟下实现高质量生成的核心在于识别最优生成时刻，这通过学习源序列与目标序列之间的映射关系完成。然而，现有方法往往依赖针对不同序列类型的任务特定启发式策略，限制了模型自适应学习源-目标映射的能力，并阻碍了多种同步任务的多任务学习探索。本文提出一种统一的段到段框架（Seg2Seg），用于同步序列生成，该框架以自适应且统一的方式学习映射关系。在同步生成过程中，模型交替执行等待源段与生成目标段，使得段成为源与目标之间的天然桥梁。为此，Seg2Seg引入潜在段作为源与目标之间的枢轴，并通过所提出的期望训练探索所有可能的源-目标映射，从而学习最优生成时刻。在多个同步生成任务上的实验表明，Seg2Seg实现了最先进的性能，并在不同任务间展现出更强的通用性。