We introduce SPEAR-TTS, a multi-speaker text-to-speech (TTS) system that can be trained with minimal supervision. By combining two types of discrete speech representations, we cast TTS as a composition of two sequence-to-sequence tasks: from text to high-level semantic tokens (akin to "reading") and from semantic tokens to low-level acoustic tokens ("speaking"). Decoupling these two tasks enables training of the "speaking" module using abundant audio-only data, and unlocks the highly efficient combination of pretraining and backtranslation to reduce the need for parallel data when training the "reading" component. To control the speaker identity, we adopt example prompting, which allows SPEAR-TTS to generalize to unseen speakers using only a short sample of 3 seconds, without any explicit speaker representation or speaker-id labels. Our experiments demonstrate that SPEAR-TTS achieves a character error rate that is competitive with state-of-the-art methods using only 15 minutes of parallel data, while matching ground-truth speech in terms of naturalness and acoustic quality, as measured in subjective tests.

翻译：我们提出SPEAR-TTS，一种可在极低监督下训练的多说话人文本转语音（TTS）系统。通过结合两种离散语音表征，我们将TTS建模为两个序列到序列任务的组合：从文本到高层语义令牌（相当于“读”）以及从语义令牌到低层声学令牌（相当于“说”）。将这两个任务解耦，使得“说”模块能够利用丰富的纯音频数据进行训练，并通过预训练与反向翻译的高效组合来降低训练“读”组件时对平行数据的需求。为控制说话人身份，我们采用示例提示，使SPEAR-TTS仅需3秒的简短样本即可泛化至未见过的说话人，无需显式说话人表征或说话人身份标签。实验表明，SPEAR-TTS仅需15分钟平行数据即可实现与最先进方法相当的字错误率，并在主观测试中达到与真实语音相近的自然度和声学质量。