Current speech LLMs bridge speech foundation models to LLMs using projection layers, training all of these components on speech instruction data. This strategy is computationally intensive and susceptible to task and prompt overfitting. We present SpeechMapper, a cost-efficient speech-to-LLM-embedding training approach that mitigates overfitting, enabling more robust and generalizable models. Our model is first pretrained without the LLM on inexpensive hardware, and then efficiently attached to the target LLM via a brief 1K-step instruction tuning (IT) stage. Through experiments on speech translation and spoken question answering, we demonstrate the versatility of SpeechMapper's pretrained block, presenting results for both task-agnostic IT, an ASR-based adaptation strategy that does not train in the target task, and task-specific IT. In task-agnostic settings, Speechmapper rivals the best instruction-following speech LLM from IWSLT25, despite never being trained on these tasks, while in task-specific settings, it outperforms this model across many datasets, despite requiring less data and compute. Overall, SpeechMapper offers a practical and scalable approach for efficient, generalizable speech-LLM integration without large-scale IT.

翻译：当前的语音大语言模型通过投影层将语音基础模型与大语言模型连接，并在语音指令数据上训练所有这些组件。该策略计算成本高昂，且易受任务和提示过拟合的影响。我们提出SpeechMapper，一种成本高效的语音到大语言模型嵌入训练方法，能够缓解过拟合，实现更鲁棒且可泛化的模型。我们的模型首先在廉价硬件上不依赖大语言模型进行预训练，随后通过一个简短的1K步指令微调阶段高效地连接到目标大语言模型。通过在语音翻译和口语问答任务上的实验，我们证明了SpeechMapper预训练模块的通用性，展示了任务无关指令微调（一种不在目标任务上训练的基于自动语音识别的适应策略）和任务特定指令微调的结果。在任务无关设置下，SpeechMapper的性能可与IWSLT25中表现最佳的指令跟随语音大语言模型相媲美，尽管从未在这些任务上训练过；而在任务特定设置下，尽管所需数据和计算量更少，它在多个数据集上均超越了该模型。总体而言，SpeechMapper为高效、可泛化的语音-大语言模型集成提供了一种实用且可扩展的途径，无需依赖大规模指令微调。