We recently developed SLM, a joint speech and language model, which fuses a pretrained foundational speech model and a large language model (LLM), while preserving the in-context learning capability intrinsic to the pretrained LLM. In this paper, we apply SLM to speech dialog applications where the dialog states are inferred directly from the audio signal. Task-oriented dialogs often contain domain-specific entities, i.e., restaurants, hotels, train stations, and city names, which are difficult to recognize, however, critical for the downstream applications. Inspired by the RAG (retrieval-augmented generation) paradigm, we propose a retrieval augmented SLM (ReSLM) that overcomes this weakness. We first train a speech retriever to retrieve text entities mentioned in the audio. The retrieved entities are then added as text inputs to the underlying SLM to bias model predictions. We evaluated ReSLM on speech MultiWoz task (DSTC-11 challenge), and found that this retrieval augmentation boosts model performance, achieving joint goal accuracy (38.6% vs 32.7%), slot error rate (20.6% vs 24.8%) and ASR word error rate (5.5% vs 6.7%). While demonstrated on dialog state tracking, our approach is broadly applicable to other speech tasks requiring contextual information or domain-specific entities, such as contextual ASR with biasing capability.

翻译：我们近期开发了SLM（Speech Language Model），一种融合预训练基础语音模型与大语言模型（LLM）的联合语音语言模型，同时保留了预训练LLM固有的上下文学习能力。本文将SLM应用于口语对话场景，其中对话状态直接从音频信号中推断。任务导向型对话通常包含领域特定实体（如餐厅、酒店、火车站及城市名称等），这些实体难以识别但对下游应用至关重要。受RAG（检索增强生成）范式启发，我们提出一种检索增强型SLM（ReSLM）以克服这一缺陷。首先训练语音检索器来提取音频中提及的文本实体，随后将检索到的实体作为文本输入注入底层SLM，以偏置模型预测。我们在口语MultiWoz任务（DSTC-11挑战赛）上评估了ReSLM，发现该检索增强技术有效提升了模型性能：联合目标准确率（38.6% vs 32.7%）、槽位错误率（20.6% vs 24.8%）及ASR词错误率（5.5% vs 6.7%）均获得改善。尽管本研究聚焦于对话状态追踪领域，但该方法可广泛适用于其他需要上下文信息或领域特定实体的语音任务，例如具备偏置能力的上下文相关ASR系统。