Simultaneous sequence generation is a pivotal task for real-time scenarios, such as streaming speech recognition, simultaneous machine translation and simultaneous speech translation, where the target sequence is generated while receiving the source sequence. The crux of achieving high-quality generation with low latency lies in identifying the optimal moments for generating, accomplished by learning a mapping between the source and target sequences. However, existing methods often rely on task-specific heuristics for different sequence types, limiting the model's capacity to adaptively learn the source-target mapping and hindering the exploration of multi-task learning for various simultaneous tasks. In this paper, we propose a unified segment-to-segment framework (Seg2Seg) for simultaneous sequence generation, which learns the mapping in an adaptive and unified manner. During the process of simultaneous generation, the model alternates between waiting for a source segment and generating a target segment, making the segment serve as the natural bridge between the source and target. To accomplish this, Seg2Seg introduces a latent segment as the pivot between source to target and explores all potential source-target mappings via the proposed expectation training, thereby learning the optimal moments for generating. Experiments on multiple simultaneous generation tasks demonstrate that Seg2Seg achieves state-of-the-art performance and exhibits better generality across various tasks.

翻译：同步序列生成是实时场景中的关键任务，例如流式语音识别、同步机器翻译和同步语音翻译，这些场景需要在接收源序列的同时生成目标序列。实现高质量低延迟生成的关键在于确定生成的最佳时机，这通过学习源序列与目标序列之间的映射来完成。然而，现有方法通常依赖于针对不同序列类型的特定任务启发式规则，限制了模型自适应学习源-目标映射的能力，并阻碍了对多种同步任务的多任务学习的探索。在本文中，我们提出了一种统一的同步序列生成分段到分段框架（Seg2Seg），该框架以自适应且统一的方式学习映射。在同步生成过程中，模型交替进行等待源分段和生成目标分段，使得分段成为源与目标之间的自然桥梁。为此，Seg2Seg引入了一个潜在分段作为源到目标的枢轴，并通过所提出的期望训练探索所有可能的源-目标映射，从而学习生成的最佳时机。在多个同步生成任务上的实验表明，Seg2Seg实现了最先进的性能，并在不同任务间展现出更好的通用性。