Dialogue serves as the most natural manner of human-computer interaction (HCI). Recent advancements in speech language models (SLM) have significantly enhanced speech-based conversational AI. However, these models are limited to turn-based conversation, lacking the ability to interact with humans in real-time spoken scenarios, for example, being interrupted when the generated content is not satisfactory. To address these limitations, we explore full duplex modeling (FDM) in interactive speech language models (iSLM), focusing on enhancing real-time interaction and, more explicitly, exploring the quintessential ability of interruption. We introduce a novel model design, namely listening-while-speaking language model (LSLM), an end-to-end system equipped with both listening and speaking channels. Our LSLM employs a token-based decoder-only TTS for speech generation and a streaming self-supervised learning (SSL) encoder for real-time audio input. LSLM fuses both channels for autoregressive generation and detects turn-taking in real time. Three fusion strategies -- early fusion, middle fusion, and late fusion -- are explored, with middle fusion achieving an optimal balance between speech generation and real-time interaction. Two experimental settings, command-based FDM and voice-based FDM, demonstrate LSLM's robustness to noise and sensitivity to diverse instructions. Our results highlight LSLM's capability to achieve duplex communication with minimal impact on existing systems. This study aims to advance the development of interactive speech dialogue systems, enhancing their applicability in real-world contexts.

翻译：对话是人机交互（HCI）最自然的方式。近期语音语言模型（SLM）的进展显著提升了基于语音的对话式人工智能能力。然而，这些模型仅限于轮次式对话，缺乏在实时语音场景中与人类交互的能力，例如在生成内容不令人满意时被用户打断。为应对这些局限，我们探索了交互式语音语言模型（iSLM）中的全双工建模（FDM），重点增强实时交互能力，并更明确地探究其核心的打断能力。我们提出了一种新颖的模型设计，即边听边说语言模型（LSLM），这是一个配备听与说双通道的端到端系统。我们的LSLM采用基于令牌的仅解码器TTS进行语音生成，并利用流式自监督学习（SSL）编码器处理实时音频输入。LSLM融合双通道进行自回归生成，并实时检测话轮转换。我们探索了三种融合策略——早期融合、中期融合与晚期融合，其中中期融合在语音生成与实时交互间取得了最佳平衡。在基于指令的FDM和基于语音的FDM两种实验设置中，LSLM均表现出对噪声的鲁棒性及对多样化指令的敏感性。我们的结果突显了LSLM能够以对现有系统影响最小的方式实现双工通信。本研究旨在推动交互式语音对话系统的发展，提升其在现实场景中的适用性。