We present a joint Speech and Language Model (SLM), a multitask, multilingual, and dual-modal model that takes advantage of pretrained foundational speech and language models. SLM freezes the pretrained foundation models to maximally preserves their capabilities, and only trains a simple adapter with just 1\% (156M) of the foundation models' parameters. This adaptation not only leads SLM to achieve strong performance on conventional tasks such as speech recognition (ASR) and speech translation (AST), but also introduces the novel capability of zero-shot instruction-following for more diverse tasks: given a speech input and a text instruction, SLM is able to perform unseen generation tasks including contextual biasing ASR using real-time context, dialog generation, speech continuation, and question answering, etc. Our approach demonstrates that the representational gap between pretrained speech and language models might be narrower than one would expect, and can be bridged by a simple adaptation mechanism. As a result, SLM is not only efficient to train, but also inherits strong capabilities already acquired in foundation models of different modalities.

翻译：我们提出了一种联合语音与语言模型（SLM），这是一个多任务、多语言、双模态模型，充分利用了预训练的语音与语言基础模型。SLM冻结预训练基础模型以最大程度保留其能力，仅训练一个仅占基础模型参数1%（1.56亿）的简单适配器。该适配方法不仅使SLM在语音识别（ASR）和语音翻译（AST）等传统任务上取得优异性能，还引入了零样本指令跟随这一新颖能力，可处理更多样化的任务：给定语音输入和文本指令，SLM能够执行未见过的生成任务，包括基于实时上下文的上下文感知ASR、对话生成、语音延续和问答等。我们的方法表明，预训练语音与语言模型之间的表征鸿沟可能比预期的更窄，且可通过简单的适配机制弥合。因此，SLM不仅训练高效，还继承了不同模态基础模型已具备的强大能力。