Improving text representation has attracted much attention to achieve expressive text-to-speech (TTS). However, existing works only implicitly learn the prosody with masked token reconstruction tasks, which leads to low training efficiency and difficulty in prosody modeling. We propose CLAPSpeech, a cross-modal contrastive pre-training framework that explicitly learns the prosody variance of the same text token under different contexts. Specifically, 1) We encourage the model to connect the text context with its corresponding prosody pattern in the joint multi-modal space with the elaborate design of the encoder inputs and contrastive loss; 2) We introduce a multi-scale pre-training pipeline to capture prosody patterns in multiple levels. We show how to incorporate CLAPSpeech into existing TTS models for better prosody. Experiments on three datasets not only show that CLAPSpeech could improve the prosody prediction for existing TTS methods, but also demonstrate its generalization ability to adapt to multiple languages and multi-speaker TTS. We also deeply analyze the principle behind the performance of CLAPSpeech. Ablation studies demonstrate the necessity of each component in our method. Source code and audio samples are available at https://clapspeech.github.io.

翻译：改善文本表示以实现富有表现力的文本到语音（TTS）近年来备受关注。然而，现有方法仅通过掩码标记重构任务隐式学习韵律，导致训练效率低且韵律建模困难。我们提出CLAPSpeech，一种跨模态对比预训练框架，能够显式学习相同文本标记在不同上下文中的韵律变化。具体而言：1）通过精心设计的编码器输入和对比损失，我们促使模型在联合多模态空间中建立文本上下文与其相应韵律模式之间的关联；2）我们引入多尺度预训练流程以捕获多级韵律模式。我们展示了如何将CLAPSpeech集成到现有TTS模型中以获得更优韵律。在三个数据集上的实验不仅表明CLAPSpeech能够改进现有TTS方法的韵律预测能力，还证明了其适应多语言和多说话人TTS的泛化能力。我们深入分析了CLAPSpeech性能背后的原理。消融研究验证了该方法中各组件的必要性。源代码和音频样本可于https://clapspeech.github.io获取。