Multi-modal large language models have garnered significant interest recently. Though, most of the works focus on vision-language multi-modal models providing strong capabilities in following vision-and-language instructions. However, we claim that speech is also an important modality through which humans interact with the world. Hence, it is crucial for a general-purpose assistant to be able to follow multi-modal speech-and-language instructions. In this work, we propose Large Language and Speech Model (LLaSM). LLaSM is an end-to-end trained large multi-modal speech-language model with cross-modal conversational abilities, capable of following speech-and-language instructions. Our early experiments show that LLaSM demonstrates a more convenient and natural way for humans to interact with artificial intelligence. Specifically, we also release a large Speech Instruction Following dataset LLaSM-Audio-Instructions. Code and demo are available at https://github.com/LinkSoul-AI/LLaSM and https://huggingface.co/spaces/LinkSoul/LLaSM. The LLaSM-Audio-Instructions dataset is available at https://huggingface.co/datasets/LinkSoul/LLaSM-Audio-Instructions.

翻译：多模态大型语言模型近期引起了广泛关注。然而，现有工作大多聚焦于视觉-语言多模态模型，虽能有效遵循视觉与语言指令，但我们认为语音同样是人类与世界交互的重要模态。因此，通用型助手必须具备遵循多模态语音-语言指令的能力。本文提出大型语言与语音模型（LLaSM），这是一种端到端训练的大型多模态语音-语言模型，具备跨模态对话能力，能够执行语音与语言指令。初步实验表明，LLaSM为人类与人工智能的交互提供了更便捷、自然的方式。特别地，我们还发布了大型语音指令遵循数据集LLaSM-Audio-Instructions。代码和演示可访问https://github.com/LinkSoul-AI/LLaSM 及 https://huggingface.co/spaces/LinkSoul/LLaSM。LLaSM-Audio-Instructions数据集可于https://huggingface.co/datasets/LinkSoul/LLaSM-Audio-Instructions 获取。