We propose an approach for continuous prediction of turn-taking and backchanneling locations in spoken dialogue by fusing a neural acoustic model with a large language model (LLM). Experiments on the Switchboard human-human conversation dataset demonstrate that our approach consistently outperforms the baseline models with single modality. We also develop a novel multi-task instruction fine-tuning strategy to further benefit from LLM-encoded knowledge for understanding the tasks and conversational contexts, leading to additional improvements. Our approach demonstrates the potential of combined LLMs and acoustic models for a more natural and conversational interaction between humans and speech-enabled AI agents.

翻译：我们提出一种通过融合神经声学模型与大语言模型，实现口语对话中轮流说话与回馈位置连续预测的方法。在Switchboard人-人对话数据集上的实验表明，我们的方法始终优于单一模态的基线模型。我们还开发了一种创新的多任务指令微调策略，以进一步利用大语言模型编码的知识来理解任务与对话上下文，从而带来额外性能提升。该方法展示了将大语言模型与声学模型相结合，在人类与语音交互AI agent之间实现更自然对话互动的潜力。