The recent surge in popularity of diffusion models for image generation has brought new attention to the potential of these models in other areas of media generation. One area that has yet to be fully explored is the application of diffusion models to audio generation. Audio generation requires an understanding of multiple aspects, such as the temporal dimension, long term structure, multiple layers of overlapping sounds, and the nuances that only trained listeners can detect. In this work, we investigate the potential of diffusion models for audio generation. We propose a set of models to tackle multiple aspects, including a new method for text-conditional latent audio diffusion with stacked 1D U-Nets, that can generate multiple minutes of music from a textual description. For each model, we make an effort to maintain reasonable inference speed, targeting real-time on a single consumer GPU. In addition to trained models, we provide a collection of open source libraries with the hope of simplifying future work in the field. Samples can be found at https://bit.ly/audio-diffusion. Codes are at https://github.com/archinetai/audio-diffusion-pytorch.

翻译：近期扩散模型在图像生成领域的广泛应用，引发了学界对其在媒体生成其他领域潜力的重新关注。其中一个尚未充分探索的方向是扩散模型在音频生成中的应用。音频生成需要理解多个维度，包括时间维度、长程结构、多层重叠声音以及仅训练有素的听众才能察觉的细微差别。本文研究了扩散模型在音频生成中的潜力，提出了一套涵盖多个层面的模型体系，包括一种基于堆叠一维U-Net的文本条件潜变量音频扩散新方法，该方法能够根据文本描述生成数分钟的音乐。我们致力于在保持合理推理速度的前提下针对每个模型进行优化，目标是在单张消费级GPU上实现实时生成。除训练好的模型外，我们还提供了一系列开源库，以期望简化该领域的后续研究。示例音频可访问https://bit.ly/audio-diffusion，代码开源于https://github.com/archinetai/audio-diffusion-pytorch。