Large language models (LLMs) have demonstrated promising performance in both automatic speech recognition (ASR) and text-to-speech (TTS) systems, gradually becoming the mainstream approach. However, most current approaches address these tasks separately rather than through a unified framework. This work aims to integrate these two tasks into one unified model. Although discrete speech tokenization enables joint modeling, its inherent information loss limits performance in both recognition and generation. In this work, we present UniVoice, a unified LLM framework through continuous representations that seamlessly integrates speech recognition and synthesis within a single model. Our approach combines the strengths of autoregressive modeling for speech recognition with flow matching for high-quality generation. To mitigate the inherent divergence between autoregressive and flow-matching models, we further design a dual attention mechanism, which switches between a causal mask for recognition and a bidirectional attention mask for synthesis. Furthermore, the proposed text-prefix-conditioned speech infilling method enables high-fidelity zero-shot voice cloning. Experimental results demonstrate that our method can achieve or exceed current single-task modeling methods in both ASR and zero-shot TTS tasks. This work explores new possibilities for end-to-end speech understanding and generation. Code is available at https://github.com/gwh22/UniVoice.

翻译：大型语言模型（LLMs）在自动语音识别（ASR）和文本转语音（TTS）系统中均展现出卓越性能，正逐步成为主流方法。然而，当前多数研究将这两类任务分开处理，而非采用统一框架。本文旨在将两种任务整合至单一模型。尽管离散语音标记化技术可实现联合建模，但其固有的信息损失限制了识别与生成任务的性能。为此，我们提出UniVoice——一种基于连续表示的统一LLM框架，能够无缝集成语音识别与合成功能。该方法融合了自回归建模在语音识别中的优势与流匹配在高品质生成中的能力。为缓解自回归模型与流匹配模型之间的固有差异，我们进一步设计了双重注意力机制，可在识别任务中的因果掩码与合成任务中的双向注意力掩码间动态切换。此外，所提出的文本前缀条件语音填充方法实现了高保真度的零样本语音克隆。实验结果表明，本方法在ASR与零样本TTS任务中均可达到或超越当前单任务建模方法。这项工作为端到端语音理解与生成探索了新的可能性。代码开源地址：https://github.com/gwh22/UniVoice。