Expressive text-to-speech (TTS) aims to synthesize different speaking style speech according to human's demands. Nowadays, there are two common ways to control speaking styles: (1) Pre-defining a group of speaking style and using categorical index to denote different speaking style. However, there are limitations in the diversity of expressiveness, as these models can only generate the pre-defined styles. (2) Using reference speech as style input, which results in a problem that the extracted style information is not intuitive or interpretable. In this study, we attempt to use natural language as style prompt to control the styles in the synthetic speech, \textit{e.g.}, ``Sigh tone in full of sad mood with some helpless feeling". Considering that there is no existing TTS corpus which is proper to benchmark this novel task, we first construct a speech corpus, whose speech samples are annotated with not only content transcriptions but also style descriptions in natural language. Then we propose an expressive TTS model, named as InstructTTS, which is novel in the sense of following aspects: (1) We fully take the advantage of self-supervised learning and cross-modal metric learning, and propose a novel three-stage training procedure to obtain a robust sentence embedding model, which can effectively capture semantic information from the style prompts and control the speaking style in the generated speech. (2) We propose to model acoustic features in discrete latent space and train a novel discrete diffusion probabilistic model to generate vector-quantized (VQ) acoustic tokens rather than the commonly-used mel spectrogram. (3) We jointly apply mutual information (MI) estimation and minimization during acoustic model training to minimize style-speaker and style-content MI, avoiding possible content and speaker information leakage from the style prompt.

翻译：表现性文本到语音合成（TTS）旨在根据人类需求合成不同说话风格的语音。当前，控制说话风格的常见方法有两种：（1）预定义一组说话风格，并使用类别索引表示不同风格。然而，由于这类模型仅能生成预定义风格，其表现多样性存在局限。（2）使用参考语音作为风格输入，但这会导致提取的风格信息缺乏直观性与可解释性。本研究尝试将自然语言作为风格提示来控制合成语音的风格，例如“充满悲伤情绪且带有无奈感的叹息语调”。考虑到现有TTS语料库均不适用于基准测试这一新任务，我们首先构建了一个语音语料库，其中的语音样本不仅标注了内容转录文本，还标注了自然语言风格描述。接着，我们提出一种名为InstructTTS的表现性TTS模型，其创新性体现在以下方面：（1）充分利用自监督学习与跨模态度量学习的优势，提出一种新颖的三阶段训练流程，以获取鲁棒的句子嵌入模型，该模型能有效从风格提示中捕获语义信息并控制生成语音的说话风格；（2）提出在离散潜空间中建模声学特征，并训练一种新颖的离散扩散概率模型来生成向量量化（VQ）声学标记，而非常用的梅尔频谱图；（3）在声学模型训练过程中联合应用互信息（MI）估计与最小化，以降低风格-说话人及风格-内容间的MI，避免风格提示中潜在的内容与说话人信息泄露。