Large audio-language models (LALMs), built upon powerful Large Language Models (LLMs), have exhibited remarkable audio comprehension and reasoning capabilities. However, the training of LALMs demands a large corpus of audio-language pairs, which requires substantial costs in both data collection and training resources. In this paper, we propose \textbf{MATS}, an audio-language multimodal LLM designed to handle \textbf{M}ultiple \textbf{A}udio task using solely \textbf{T}ext-only \textbf{S}upervision. By leveraging pre-trained audio-language alignment models such as CLAP, we develop a text-only training strategy that projects the shared audio-language latent space into LLM latent space, endowing the LLM with audio comprehension capabilities without relying on audio data during training. To further bridge the modality gap between audio and language embeddings within CLAP, we propose the \textbf{S}trongly-rel\textbf{a}ted \textbf{n}oisy \textbf{t}ext with \textbf{a}udio (\textbf{Santa}) mechanism. Santa maps audio embeddings into CLAP language embedding space while preserving essential information from the audio input. Extensive experiments demonstrate that MATS, despite being trained exclusively on text data, achieves competitive performance compared to recent LALMs trained on large-scale audio-language pairs. The code is publicly available in \href{https://github.com/wangwen-banban/MATS}{https://github.com/wangwen-banban/MATS}.

翻译：大型音频语言模型（LALMs）基于强大的大语言模型（LLMs）构建，已展现出卓越的音频理解与推理能力。然而，LALMs的训练需要大规模的音频-语言对语料库，这在数据收集和训练资源方面均需耗费巨大成本。本文提出 \textbf{MATS}，一种音频语言多模态大语言模型，旨在仅使用\textbf{仅文本监督}处理\textbf{多}种\textbf{音频}任务。通过利用预训练的音频-语言对齐模型（如CLAP），我们开发了一种仅文本训练策略，该策略将共享的音频-语言潜在空间投影至LLM潜在空间，从而赋予LLM音频理解能力，而无需在训练过程中依赖音频数据。为了进一步弥合CLAP内部音频嵌入与语言嵌入之间的模态鸿沟，我们提出了\textbf{强相关噪声文本与音频}（\textbf{Santa}）机制。Santa将音频嵌入映射到CLAP语言嵌入空间，同时保留音频输入中的关键信息。大量实验表明，MATS尽管仅使用文本数据进行训练，其性能仍可与近期基于大规模音频-语言对训练的LALMs相媲美。代码已在 \href{https://github.com/wangwen-banban/MATS}{https://github.com/wangwen-banban/MATS} 公开。