The utilization of discrete speech tokens, divided into semantic tokens and acoustic tokens, has been proven superior to traditional acoustic feature mel-spectrograms in terms of naturalness and robustness for text-to-speech (TTS) synthesis. Recent popular models, such as VALL-E and SPEAR-TTS, allow zero-shot speaker adaptation through auto-regressive (AR) continuation of acoustic tokens extracted from a short speech prompt. However, these AR models are restricted to generate speech only in a left-to-right direction, making them unsuitable for speech editing where both preceding and following contexts are provided. Furthermore, these models rely on acoustic tokens, which have audio quality limitations imposed by the performance of audio codec models. In this study, we propose a unified context-aware TTS framework called UniCATS, which is capable of both speech continuation and editing. UniCATS comprises two components, an acoustic model CTX-txt2vec and a vocoder CTX-vec2wav. CTX-txt2vec employs contextual VQ-diffusion to predict semantic tokens from the input text, enabling it to incorporate the semantic context and maintain seamless concatenation with the surrounding context. Following that, CTX-vec2wav utilizes contextual vocoding to convert these semantic tokens into waveforms, taking into consideration the acoustic context. Our experimental results demonstrate that CTX-vec2wav outperforms HifiGAN and AudioLM in terms of speech resynthesis from semantic tokens. Moreover, we show that UniCATS achieves state-of-the-art performance in both speech continuation and editing.

翻译：离散语音令牌（分为语义令牌和声学令牌）的运用已被证明在文本转语音合成的自然度和鲁棒性方面优于传统声学特征梅尔频谱图。近期热门模型（如VALL-E和SPEAR-TTS）通过自回归延续从短语音提示中提取的声学令牌，实现了零样本说话人自适应。然而，这些自回归模型仅能生成从左到右方向的语音，使其不适用于需同时提供前后语境的语音编辑任务。此外，这些模型依赖声学令牌，其音频质量受限于音频编解码模型的性能。本研究提出一种名为UniCATS的统一上下文感知文本转语音框架，能够同时实现语音延续与编辑。UniCATS包含两个组件：声学模型CTX-txt2vec和声码器CTX-vec2wav。CTX-txt2vec利用上下文VQ扩散从输入文本预测语义令牌，从而整合语义上下文并保持与周围语境的无缝衔接。随后，CTX-vec2wav通过考虑声学上下文的上下文声码化，将这些语义令牌转换为波形。实验结果表明，CTX-vec2wav在语义令牌语音重合成任务中优于HifiGAN和AudioLM。此外，我们证明UniCATS在语音延续与编辑任务中均达到了最先进性能。