Zero-shot text-to-speech (TTS) aims to synthesize voices with unseen speech prompts, which significantly reduces the data and computation requirements for voice cloning by skipping the fine-tuning process. However, the prompting mechanisms of zero-shot TTS still face challenges in the following aspects: 1) previous works of zero-shot TTS are typically trained with single-sentence prompts, which significantly restricts their performance when the data is relatively sufficient during the inference stage. 2) The prosodic information in prompts is highly coupled with timbre, making it untransferable to each other. This paper introduces Mega-TTS 2, a generic prompting mechanism for zero-shot TTS, to tackle the aforementioned challenges. Specifically, we design a powerful acoustic autoencoder that separately encodes the prosody and timbre information into the compressed latent space while providing high-quality reconstructions. Then, we propose a multi-reference timbre encoder and a prosody latent language model (P-LLM) to extract useful information from multi-sentence prompts. We further leverage the probabilities derived from multiple P-LLM outputs to produce transferable and controllable prosody. Experimental results demonstrate that Mega-TTS 2 could not only synthesize identity-preserving speech with a short prompt of an unseen speaker from arbitrary sources but consistently outperform the fine-tuning method when the volume of data ranges from 10 seconds to 5 minutes. Furthermore, our method enables to transfer various speaking styles to the target timbre in a fine-grained and controlled manner. Audio samples can be found in https://boostprompt.github.io/boostprompt/.

翻译：零样本文本转语音（TTS）旨在利用未见过的语音提示合成声音，通过省去微调过程，显著降低语音克隆所需的数据量和计算资源。然而，零样本TTS的提示机制在以下方面仍面临挑战：1）以往的零样本TTS方法通常使用单句提示进行训练，这在推理阶段数据相对充足时严重限制了其性能。2）提示中的韵律信息与音色高度耦合，导致彼此无法迁移。本文提出Mega-TTS 2，一种通用的零样本TTS提示机制，以应对上述挑战。具体而言，我们设计了一个强大的声学自编码器，将韵律和音色信息分别编码到压缩的潜在空间中，同时保证高质量的重建。接着，我们提出多参考音色编码器和韵律潜在语言模型（P-LLM），以从多句提示中提取有用信息。进一步利用多个P-LLM输出的概率，生成可迁移且可控的韵律。实验结果表明，Mega-TTS 2不仅能利用任意来源的未见说话者的短提示合成保留音色的语音，而且当数据量从10秒到5分钟变化时，其性能始终优于微调方法。此外，我们的方法能以细粒度和可控的方式将多种说话风格迁移到目标音色。音频样本可在https://boostprompt.github.io/boostprompt/获取。