Recently, there has been a growing interest in the field of controllable Text-to-Speech (TTS). While previous studies have relied on users providing specific style factor values based on acoustic knowledge or selecting reference speeches that meet certain requirements, generating speech solely from natural text prompts has emerged as a new challenge for researchers. This challenge arises due to the scarcity of high-quality speech datasets with natural text style prompt and the absence of advanced text-controllable TTS models. In light of this, 1) we propose TextrolSpeech, which is the first large-scale speech emotion dataset annotated with rich text attributes. The dataset comprises 236,220 pairs of style prompt in natural text descriptions with five style factors and corresponding speech samples. Through iterative experimentation, we introduce a multi-stage prompt programming approach that effectively utilizes the GPT model for generating natural style descriptions in large volumes. 2) Furthermore, to address the need for generating audio with greater style diversity, we propose an efficient architecture called Salle. This architecture treats text controllable TTS as a language model task, utilizing audio codec codes as an intermediate representation to replace the conventional mel-spectrogram. Finally, we successfully demonstrate the ability of the proposed model by showing a comparable performance in the controllable TTS task. Audio samples are available at https://sall-e.github.io/

翻译：近年来，可控文本转语音（TTS）领域引起了广泛关注。以往的研究依赖用户根据声学知识提供具体的风格因子数值，或选取满足特定要求的参考语音，而仅凭自然文本提示生成语音已成为研究人员面临的新挑战。这一挑战源于两方面：一是缺乏带有自然文本风格提示的高质量语音数据集，二是缺乏先进的文本可控TTS模型。针对此问题，本文提出：1）TextrolSpeech——首个以丰富文本属性标注的大规模语音情感数据集。该数据集包含236,220对具有五种风格因子的自然文本风格描述及对应的语音样本；通过迭代实验，我们引入一种多阶段提示编程方法，有效利用GPT模型实现大规模自然风格描述的生成。2）此外，为满足生成更高风格多样性音频的需求，我们提出一种名为Salle的高效架构。该架构将文本可控TTS视为语言模型任务，利用音频编解码码作为中间表征替代传统梅尔频谱。最终，我们通过展示可控TTS任务中的可比性能，成功验证了所提模型的能力。音频样本见https://sall-e.github.io/