Simultaneous sequence generation is a pivotal task for real-time scenarios, such as streaming speech recognition, simultaneous machine translation and simultaneous speech translation, where the target sequence is generated while receiving the source sequence. The crux of achieving high-quality generation with low latency lies in identifying the optimal moments for generating, accomplished by learning a mapping between the source and target sequences. However, existing methods often rely on task-specific heuristics for different sequence types, limiting the model's capacity to adaptively learn the source-target mapping and hindering the exploration of multi-task learning for various simultaneous tasks. In this paper, we propose a unified segment-to-segment framework (Seg2Seg) for simultaneous sequence generation, which learns the mapping in an adaptive and unified manner. During the process of simultaneous generation, the model alternates between waiting for a source segment and generating a target segment, making the segment serve as the natural bridge between the source and target. To accomplish this, Seg2Seg introduces a latent segment as the pivot between source to target and explores all potential source-target mappings via the proposed expectation training, thereby learning the optimal moments for generating. Experiments on multiple simultaneous generation tasks demonstrate that Seg2Seg achieves state-of-the-art performance and exhibits better generality across various tasks.

翻译：同步序列生成是实时场景中的关键任务，例如流式语音识别、同步机器翻译和同步语音翻译，这些场景要求在接收源序列的同时生成目标序列。实现高质量且低延迟生成的核心在于识别最佳的生成时机，这通过学习源序列与目标序列之间的映射关系来完成。然而，现有方法通常依赖针对不同序列类型的任务特定启发式规则，限制了模型自适应学习源-目标映射的能力，并阻碍了各类同步任务的多任务学习探索。本文提出了一种统一的段到段框架（Seg2Seg），用于同步序列生成，该框架以自适应且统一的方式学习映射关系。在同步生成过程中，模型交替进行等待源段和生成目标段，使得段成为源与目标之间的自然桥梁。为实现此目标，Seg2Seg引入了一个潜在段作为源与目标之间的枢轴，并通过所提出的期望训练探索所有可能的源-目标映射，从而学习最佳的生成时机。在多个同步生成任务上的实验表明，Seg2Seg达到了最先进的性能，并在不同任务间展现出更好的泛化能力。