Recent research shows a big convergence in model architecture, training objectives, and inference methods across various tasks for different modalities. In this paper, we propose VioLA, a single auto-regressive Transformer decoder-only network that unifies various cross-modal tasks involving speech and text, such as speech-to-text, text-to-text, text-to-speech, and speech-to-speech tasks, as a conditional codec language model task via multi-task learning framework. To accomplish this, we first convert all the speech utterances to discrete tokens (similar to the textual data) using an offline neural codec encoder. In such a way, all these tasks are converted to token-based sequence conversion problems, which can be naturally handled with one conditional language model. We further integrate task IDs (TID) and language IDs (LID) into the proposed model to enhance the modeling capability of handling different languages and tasks. Experimental results demonstrate that the proposed VioLA model can support both single-modal and cross-modal tasks well, and the decoder-only model achieves a comparable and even better performance than the strong baselines.

翻译：近期研究表明，不同模态任务的模型架构、训练目标及推理方法呈现高度趋同趋势。本文提出VioLA——一种仅含解码器的自回归Transformer网络，通过多任务学习框架将语音与文本间的各类跨模态任务（如语音转文本、文本转文本、文本转语音及语音转语音）统一为条件式编解码语言模型任务。具体而言，我们首先利用离线神经编解码器将所有语音发声转换为离散化词元（类似文本数据），从而将这些任务转化为可基于单个条件语言模型自然处理的词元序列转换问题。此外，我们在模型中集成任务标识与语言标识，以增强其处理多语言多任务的建模能力。实验结果表明，VioLA模型能有效支持单模态与跨模态任务，且该仅含解码器的模型在性能上达到甚至超越了强基线方法。