For fine-grained generation and recognition tasks such as minimally-supervised text-to-speech (TTS), voice conversion (VC), and automatic speech recognition (ASR), the intermediate representations extracted from speech should serve as a "bridge" between text and acoustic information, containing information from both modalities. The semantic content is emphasized, while the paralinguistic information such as speaker identity and acoustic details should be de-emphasized. However, existing methods for extracting fine-grained intermediate representations from speech suffer from issues of excessive redundancy and dimension explosion. Contrastive learning is a good method for modeling intermediate representations from two modalities. However, existing contrastive learning methods in the audio field focus on extracting global descriptive information for downstream audio classification tasks, making them unsuitable for TTS, VC, and ASR tasks. To address these issues, we propose a method named "Contrastive Token-Acoustic Pretraining (CTAP)", which uses two encoders to bring phoneme and speech into a joint multimodal space, learning how to connect phoneme and speech at the frame level. The CTAP model is trained on 210k speech and phoneme text pairs, achieving minimally-supervised TTS, VC, and ASR. The proposed CTAP method offers a promising solution for fine-grained generation and recognition downstream tasks in speech processing.

翻译：摘要：对于细粒度生成与识别任务，如最小监督文本转语音（TTS）、语音转换（VC）和自动语音识别（ASR），从语音中提取的中间表示应在文本与声学信息之间起到“桥梁”作用，同时包含两种模态的信息。语义内容应被强调，而说话人身份和声学细节等副语言信息则应被弱化。然而，现有从语音中提取细粒度中间表示的方法存在过度冗余和维度爆炸的问题。对比学习是建模两种模态间中间表示的有效方法，但当前音频领域的对比学习方法主要聚焦于提取全局描述性信息以服务于下游音频分类任务，不适用于TTS、VC和ASR任务。为解决这些问题，我们提出一种名为“对比式令牌-声学预训练（CTAP）”的方法，该方法使用两个编码器将音素与语音映射至联合多模态空间，学习在帧级连接音素与语音。CTAP模型基于21万条语音-音素文本对进行训练，实现了最小监督的TTS、VC和ASR。所提出的CTAP方法为语音处理中的细粒度生成与识别下游任务提供了一种有前景的解决方案。