Large Language Models (LLMs) demonstrate impressive capabilities, yet interaction with these models is mostly facilitated through text. Using Text-To-Speech to synthesize LLM outputs typically results in notable latency, which is impractical for fluent voice conversations. We propose LLM2Speech, an architecture to synthesize speech while text is being generated by an LLM which yields significant latency reduction. LLM2Speech mimics the predictions of a non-streaming teacher model while limiting the exposure to future context in order to enable streaming. It exploits the hidden embeddings of the LLM, a by-product of the text generation that contains informative semantic context. Experimental results show that LLM2Speech maintains the teacher's quality while reducing the latency to enable natural conversations.

翻译：大语言模型展现出令人印象深刻的能力，然而与这些模型的交互目前主要通过文本实现。使用文本转语音合成大语言模型的输出通常会产生显著的延迟，这不利于流畅的语音对话。我们提出LLM2Speech架构，该架构能在LLM生成文本的同时合成语音，从而大幅降低延迟。LLM2Speech在限制未来上下文暴露以实现流式处理的同时，模拟非流式教师模型的预测结果。它利用LLM的隐藏嵌入——文本生成的副产品，其中包含丰富的语义上下文信息。实验结果表明，LLM2Speech在保持教师模型质量的同时，能通过降低延迟实现自然的对话效果。