Scaling text-to-speech (TTS) to large-scale, multi-speaker, and in-the-wild datasets is important to capture the diversity in human speech such as speaker identities, prosodies, and styles (e.g., singing). Current large TTS systems usually quantize speech into discrete tokens and use language models to generate these tokens one by one, which suffer from unstable prosody, word skipping/repeating issue, and poor voice quality. In this paper, we develop NaturalSpeech 2, a TTS system that leverages a neural audio codec with residual vector quantizers to get the quantized latent vectors and uses a diffusion model to generate these latent vectors conditioned on text input. To enhance the zero-shot capability that is important to achieve diverse speech synthesis, we design a speech prompting mechanism to facilitate in-context learning in the diffusion model and the duration/pitch predictor. We scale NaturalSpeech 2 to large-scale datasets with 44K hours of speech and singing data and evaluate its voice quality on unseen speakers. NaturalSpeech 2 outperforms previous TTS systems by a large margin in terms of prosody/timbre similarity, robustness, and voice quality in a zero-shot setting, and performs novel zero-shot singing synthesis with only a speech prompt. Audio samples are available at https://speechresearch.github.io/naturalspeech2.

翻译：将文本转语音（TTS）扩展到大规模、多说话人及野外数据集，对于捕捉人类语音的多样性（如说话人身份、韵律和歌唱等风格）至关重要。当前大型TTS系统通常将语音量化为离散标记，并利用语言模型逐标记生成，但存在韵律不稳定、单词跳转/重复问题以及音质不佳等缺陷。本文提出的NaturalSpeech 2 TTS系统，采用带残差矢量量化的神经音频编解码器获取量化潜向量，并借助扩散模型基于文本输入生成这些潜向量。为增强实现多样化语音合成所需的关键零样本能力，我们设计了语音提示机制，以促进扩散模型及时长/音高预测器中的上下文学习。我们将NaturalSpeech 2扩展至包含44K小时语音和歌唱数据的大规模数据集，并在未见说话人上评估其音质。在零样本场景下，NaturalSpeech 2在韵律/音色相似度、鲁棒性和音质方面显著优于先前TTS系统，并仅凭语音提示即可实现新颖的零样本歌唱合成。音频样本参见https://speechresearch.github.io/naturalspeech2。