Audio is an essential part of our life, but creating it often requires expertise and is time-consuming. Research communities have made great progress over the past year advancing the performance of large scale audio generative models for a single modality (speech, sound, or music) through adopting more powerful generative models and scaling data. However, these models lack controllability in several aspects: speech generation models cannot synthesize novel styles based on text description and are limited on domain coverage such as outdoor environments; sound generation models only provide coarse-grained control based on descriptions like "a person speaking" and would only generate mumbling human voices. This paper presents Audiobox, a unified model based on flow-matching that is capable of generating various audio modalities. We design description-based and example-based prompting to enhance controllability and unify speech and sound generation paradigms. We allow transcript, vocal, and other audio styles to be controlled independently when generating speech. To improve model generalization with limited labels, we adapt a self-supervised infilling objective to pre-train on large quantities of unlabeled audio. Audiobox sets new benchmarks on speech and sound generation (0.745 similarity on Librispeech for zero-shot TTS; 0.77 FAD on AudioCaps for text-to-sound) and unlocks new methods for generating audio with novel vocal and acoustic styles. We further integrate Bespoke Solvers, which speeds up generation by over 25 times compared to the default ODE solver for flow-matching, without loss of performance on several tasks. Our demo is available at https://audiobox.metademolab.com/

翻译：音频是生活中不可或缺的组成部分，但其创作过程常需专业知识且耗时费力。过去一年中，研究界通过采用更强的生成模型并扩展数据规模，在单一模态（语音、音效或音乐）的大规模音频生成模型性能上取得了显著进展。然而，这些模型在可控性方面仍存在不足：语音生成模型无法基于文本描述合成新风格，且受限于户外环境等场景覆盖范围；音效生成模型仅能提供基于“人声说话”等描述的粗粒度控制，且只能生成含糊不清的人声。本文提出Audiobox——一种基于流匹配的统一模型，可生成多种音频模态。我们设计了基于描述与基于示例的提示机制以增强可控性，并统一了语音与音效生成范式。在语音生成过程中，我们支持对转录文本、音色及其他音频风格进行独立控制。为提升有限标注数据下的模型泛化能力，我们采用自监督填充目标对大量无标注音频进行预训练。Audiobox在语音与音效生成任务上树立了新标杆（零样本TTS在Librispeech上相似度达0.745；文本到音效在AudioCaps上FAD达0.77），并解锁了生成具有新颖音色与声学风格音频的新方法。我们还集成了Bespoke Solvers，相比流匹配的默认ODE求解器，生成速度提升超过25倍，且在多项任务中性能无损。演示地址：https://audiobox.metademolab.com/