Current speech LLMs bridge speech foundation models to LLMs using projection layers, training all of these components on speech instruction data. This strategy is computationally intensive and susceptible to task and prompt overfitting. We present SpeechMapper, a cost-efficient speech-to-LLM-embedding training approach that mitigates overfitting, enabling more robust and generalizable models. Our model is first pretrained without the LLM on inexpensive hardware, and then efficiently attached to the target LLM via a brief 1K-step instruction tuning (IT) stage. Through experiments on speech translation and spoken question answering, we demonstrate the versatility of SpeechMapper's pretrained block, presenting results for both task-agnostic IT, an ASR-based adaptation strategy that does not train in the target task, and task-specific IT. In task-agnostic settings, Speechmapper rivals the best instruction-following speech LLM from IWSLT25, despite never being trained on these tasks, while in task-specific settings, it outperforms this model across many datasets, despite requiring less data and compute. Overall, SpeechMapper offers a practical and scalable approach for efficient, generalizable speech-LLM integration without large-scale IT.

翻译：当前语音大语言模型通过投影层将语音基础模型与大语言模型连接，并在语音指令数据上训练所有这些组件。该策略计算成本高昂，且易受任务和提示过拟合影响。本文提出SpeechMapper，一种成本高效的语音到大语言模型嵌入训练方法，可缓解过拟合问题，从而构建更鲁棒且泛化能力更强的模型。我们的模型首先在廉价硬件上不依赖大语言模型进行预训练，随后通过仅1K步的指令微调阶段高效连接到目标大语言模型。通过在语音翻译和口语问答任务上的实验，我们证明了SpeechMapper预训练模块的通用性：展示了任务无关指令微调（一种基于自动语音识别、无需在目标任务上训练的适配策略）与任务特定指令微调的结果。在任务无关场景中，SpeechMapper的性能媲美IWSLT25中最佳的指令跟随语音大语言模型，尽管从未在这些任务上训练；而在任务特定场景中，其在使用更少数据和算力的条件下，仍能在多个数据集上超越该模型。总体而言，SpeechMapper为高效、可泛化的语音-大语言模型集成提供了一种实用且可扩展的解决方案，无需依赖大规模指令微调。