Adapting automatic speech recognition (ASR) systems based on large language models (LLMs) to new domains using text-only data is a significant yet underexplored challenge. Standard fine-tuning of the LLM on target-domain text often disrupts the critical alignment between speech and text modalities learned by the projector, degrading performance. We introduce a novel text-only adaptation method that emulates the audio projection task by treating it as a text denoising task. Our approach thus trains the LLM to recover clean transcripts from noisy inputs. This process effectively adapts the model to a target domain while preserving cross-modal alignment. Our solution is lightweight, requiring no architectural changes or additional parameters. Extensive evaluation on two datasets demonstrates up to 22.1% relative improvement, outperforming recent state-of-the-art text-only adaptation methods.

翻译：利用纯文本数据将基于大语言模型（LLM）的自动语音识别（ASR）系统适应到新领域，是一个重要但尚未充分探索的挑战。在目标领域文本上对LLM进行标准微调通常会破坏投影器学习到的语音与文本模态间的关键对齐，从而导致性能下降。我们提出了一种新颖的纯文本适应方法，通过将音频投影任务视为文本去噪任务来模拟该过程。因此，我们的方法训练LLM从含噪声的输入中恢复干净的转录文本。这一过程在保持跨模态对齐的同时，有效地使模型适应目标领域。我们的解决方案是轻量级的，无需改变架构或增加额外参数。在两个数据集上的广泛评估表明，该方法实现了高达22.1%的相对性能提升，优于近期最先进的纯文本适应方法。