Large-scale generative models such as GPT and DALL-E have revolutionized natural language processing and computer vision research. These models not only generate high fidelity text or image outputs, but are also generalists which can solve tasks not explicitly taught. In contrast, speech generative models are still primitive in terms of scale and task generalization. In this paper, we present Voicebox, the most versatile text-guided generative model for speech at scale. Voicebox is a non-autoregressive flow-matching model trained to infill speech, given audio context and text, trained on over 50K hours of speech that are neither filtered nor enhanced. Similar to GPT, Voicebox can perform many different tasks through in-context learning, but is more flexible as it can also condition on future context. Voicebox can be used for mono or cross-lingual zero-shot text-to-speech synthesis, noise removal, content editing, style conversion, and diverse sample generation. In particular, Voicebox outperforms the state-of-the-art zero-shot TTS model VALL-E on both intelligibility (5.9% vs 1.9% word error rates) and audio similarity (0.580 vs 0.681) while being up to 20 times faster. See voicebox.metademolab.com for a demo of the model.

翻译：诸如GPT和DALL-E等大规模生成模型已彻底改变了自然语言处理与计算机视觉研究。这些模型不仅能生成高保真度的文本或图像输出，更是能解决未经显式训练任务的通才型模型。相比之下，语音生成模型在规模和任务泛化方面仍处于初级阶段。本文提出Voicebox——当前功能最全面的文本引导式大规模语音生成模型。Voicebox是一种非自回归流匹配模型，通过音频上下文与文本进行语音补全训练，其训练数据包含超过5万小时未经滤波或增强的语音。与GPT类似，Voicebox可通过情境学习完成多种不同任务，且更具灵活性——它还能利用未来上下文信息进行条件生成。Voicebox可用于单语或跨语言的零样本文本转语音合成、噪声去除、内容编辑、风格转换及多样化样本生成。特别地，Voicebox在可理解性（词错误率从5.9%降至1.9%）和音频相似度（从0.580提升至0.681）上均超越当前最先进的零样本TTS模型VALL-E，同时速度提升达20倍。模型演示参见voicebox.metademolab.com。