Expressive text-to-speech (TTS) aims to synthesize different speaking style speech according to human's demands. Nowadays, there are two common ways to control speaking styles: (1) Pre-defining a group of speaking style and using categorical index to denote different speaking style. However, there are limitations in the diversity of expressiveness, as these models can only generate the pre-defined styles. (2) Using reference speech as style input, which results in a problem that the extracted style information is not intuitive or interpretable. In this study, we attempt to use natural language as style prompt to control the styles in the synthetic speech, e.g., "Sigh tone in full of sad mood with some helpless feeling". Considering that there is no existing TTS corpus which is proper to benchmark this novel task, we first construct a speech corpus, whose speech samples are annotated with not only content transcriptions but also style descriptions in natural language. Then we propose an expressive TTS model, named as InstructTTS, which is novel in the sense of following aspects: (1) We fully take the advantage of self-supervised learning and cross-modal metric learning, and propose a novel three-stage training procedure to obtain a robust sentence embedding model, which can effectively capture semantic information from the style prompts and control the speaking style in the generated speech. (2) We propose to model acoustic features in discrete latent space and train a novel discrete diffusion probabilistic model to generate vector-quantized (VQ) acoustic tokens rather than the commonly-used mel spectrogram. (3) We jointly apply mutual information (MI) estimation and minimization during acoustic model training to minimize style-speaker and style-content MI, avoiding possible content and speaker information leakage from the style prompt.

翻译：表达性文语转换（TTS）旨在根据人类需求合成不同说话风格的语音。目前，控制说话风格主要有两种常见方法：（1）预定义一组说话风格并使用分类索引表示不同风格。然而，这类模型仅能生成预定义风格，因此表达能力多样性存在局限。（2）使用参考语音作为风格输入，导致提取的风格信息缺乏直观性与可解释性。本研究尝试使用自然语言作为风格提示来控制合成语音的风格，例如“充满悲伤情绪、带有几分无助感的叹息语调”。考虑到现有TTS语料库无法为此新任务提供基准测试，我们首先构建了一个语音语料库，其语音样本不仅标注了内容转录文本，还标注了自然语言风格描述。在此基础上，我们提出名为InstructTTS的表达性TTS模型，其创新性体现在以下方面：（1）充分利用自监督学习与跨模态度量学习，提出新颖的三阶段训练流程以获取鲁棒的句子嵌入模型，该模型能有效捕捉风格提示的语义信息并控制生成语音的说话风格。（2）提出在离散潜空间中对声学特征进行建模，并训练新型离散扩散概率模型生成矢量量化（VQ）声学标记，而非常用的梅尔频谱图。（3）在声学模型训练中联合应用互信息（MI）估计与最小化技术，以降低风格-说话人与风格-内容之间的MI，避免风格提示可能泄露内容与说话人信息。