Scaling text-to-speech (TTS) to large-scale, multi-speaker, and in-the-wild datasets is important to capture the diversity in human speech such as speaker identities, prosodies, and styles (e.g., singing). Current large TTS systems usually quantize speech into discrete tokens and use language models to generate these tokens one by one, which suffer from unstable prosody, word skipping/repeating issue, and poor voice quality. In this paper, we develop NaturalSpeech 2, a TTS system that leverages a neural audio codec with residual vector quantizers to get the quantized latent vectors and uses a diffusion model to generate these latent vectors conditioned on text input. To enhance the zero-shot capability that is important to achieve diverse speech synthesis, we design a speech prompting mechanism to facilitate in-context learning in the diffusion model and the duration/pitch predictor. We scale NaturalSpeech 2 to large-scale datasets with 44K hours of speech and singing data and evaluate its voice quality on unseen speakers. NaturalSpeech 2 outperforms previous TTS systems by a large margin in terms of prosody/timbre similarity, robustness, and voice quality in a zero-shot setting, and performs novel zero-shot singing synthesis with only a speech prompt. Audio samples are available at https://speechresearch.github.io/naturalspeech2.

翻译：将文本转语音（TTS）扩展至大规模、多说话人及野外数据集，对于捕捉人类语音的多样性（如说话人身份、韵律和风格（例如歌声））至关重要。当前大型TTS系统通常将语音量化为离散token，并利用语言模型逐个生成这些token，这会导致韵律不稳定、单词跳跃/重复问题以及较差的语音质量。本文开发了NaturalSpeech 2——一种TTS系统，该系统采用带有残差向量量化器的神经音频编解码器获取量化后的潜在向量，并使用扩散模型基于文本输入生成这些潜在向量。为增强实现多样化语音合成所必需的零样本能力，我们设计了一种语音提示机制，以促进扩散模型及时长/音高预测器中的上下文学习。我们将NaturalSpeech 2扩展到包含44K小时语音及歌声数据的大规模数据集，并评估了其在未见说话人上的语音质量。在零样本设置下，NaturalSpeech 2在韵律/音色相似度、鲁棒性和语音质量方面显著优于以往TTS系统，并实现了仅依赖语音提示的零样本歌声合成。音频样本可访问 https://speechresearch.github.io/naturalspeech2。