Recently, speech-text pre-training methods have shown remarkable success in many speech and natural language processing tasks. However, most previous pre-trained models are usually tailored for one or two specific tasks, but fail to conquer a wide range of speech-text tasks. In addition, existing speech-text pre-training methods fail to explore the contextual information within a dialogue to enrich utterance representations. In this paper, we propose Speech-text dialog Pre-training for spoken dialog understanding with ExpliCiT cRoss-Modal Alignment (SPECTRA), which is the first-ever speech-text dialog pre-training model. Concretely, to consider the temporality of speech modality, we design a novel temporal position prediction task to capture the speech-text alignment. This pre-training task aims to predict the start and end time of each textual word in the corresponding speech waveform. In addition, to learn the characteristics of spoken dialogs, we generalize a response selection task from textual dialog pre-training to speech-text dialog pre-training scenarios. Experimental results on four different downstream speech-text tasks demonstrate the superiority of SPECTRA in learning speech-text alignment and multi-turn dialog context.

翻译：近期，语音-文本预训练方法在众多语音和自然语言处理任务中取得了显著成功。然而，大多数已有的预训练模型通常针对某一或两个特定任务设计，难以广泛适用于各类语音-文本任务。此外，现有语音-文本预训练方法未能充分探索对话中的上下文信息以丰富话语表示。本文提出面向口语对话理解的语音-文本对话预训练与显式跨模态对齐模型（SPECTRA），这是首个语音-文本对话预训练模型。具体而言，为考虑语音模态的时间特性，我们设计了一种新颖的时间位置预测任务来捕捉语音-文本对齐。该预训练任务旨在预测每个文本词在对应语音波形中的起止时间。同时，为学习口语对话的特征，我们将文本对话预训练中的响应选择任务推广至语音-文本对话预训练场景。在四个不同下游语音-文本任务上的实验结果证明了SPECTRA在学习语音-文本对齐和多轮对话上下文方面的优越性。