Speech conveys more information than text, as the same word can be uttered in various voices to convey diverse information. Compared to traditional text-to-speech (TTS) methods relying on speech prompts (reference speech) for voice variability, using text prompts (descriptions) is more user-friendly since speech prompts can be hard to find or may not exist at all. TTS approaches based on the text prompt face two main challenges: 1) the one-to-many problem, where not all details about voice variability can be described in the text prompt, and 2) the limited availability of text prompt datasets, where vendors and large cost of data labeling are required to write text prompts for speech. In this work, we introduce PromptTTS 2 to address these challenges with a variation network to provide variability information of voice not captured by text prompts, and a prompt generation pipeline to utilize the large language models (LLM) to compose high quality text prompts. Specifically, the variation network predicts the representation extracted from the reference speech (which contains full information about voice variability) based on the text prompt representation. For the prompt generation pipeline, it generates text prompts for speech with a speech language understanding model to recognize voice attributes (e.g., gender, speed) from speech and a large language model to formulate text prompts based on the recognition results. Experiments on a large-scale (44K hours) speech dataset demonstrate that compared to the previous works, PromptTTS 2 generates voices more consistent with text prompts and supports the sampling of diverse voice variability, thereby offering users more choices on voice generation. Additionally, the prompt generation pipeline produces high-quality text prompts, eliminating the large labeling cost. The demo page of PromptTTS 2 is available online.

翻译：语音比文本传递更多信息，因为相同词语可通过不同嗓音传达多样化的信息。相较于依赖语音提示（参考语音）实现语音多样性的传统文本转语音（TTS）方法，使用文本提示（描述）更具用户友好性，因为语音提示可能难以获取甚至根本不存在。基于文本提示的TTS方法面临两大挑战：1）多对一问题——文本提示无法完整描述语音多样性的全部细节；2）文本提示数据集稀缺——需要供应商支持与高昂数据标注成本才能为语音编写文本提示。本研究提出PromptTTS 2应对这些挑战，通过变分网络提供文本提示未捕捉的语音多样性信息，并设计提示生成流程利用大型语言模型（LLM）构建高质量文本提示。具体而言，变分网络基于文本提示表征预测参考语音（包含完整语音多样性信息）中提取的特征。提示生成流程则通过语音语言理解模型识别语音属性（如性别、语速），并借助大型语言模型基于识别结果生成文本提示。基于大规模（44,000小时）语音数据集的实验表明，相较于先前研究，PromptTTS 2生成的语音与文本提示更一致，支持多样化语音变体采样，为用户提供更丰富的语音生成选择。此外，提示生成流程可产出高质量文本提示，消除高昂标注成本。PromptTTS 2的演示页面已在线开放。