Previous audio generation mainly focuses on specified sound classes such as speech or music, whose form and content are greatly restricted. In this paper, we go beyond specific audio generation by using natural language description as a clue to generate broad sounds. Unlike visual information, a text description is concise by its nature but has rich hidden meanings beneath, which poses a higher possibility and complexity on the audio to be generated. A Variation-Quantized GAN is used to train a codebook learning discrete representations of spectrograms. For a given text description, its pre-trained embedding is fed to a Transformer to sample codebook indices to decode a spectrogram to be further transformed into waveform by a melgan vocoder. The generated waveform has high quality and fidelity while excellently corresponding to the given text. Experiments show that our proposed method is capable of generating natural, vivid audios, achieving superb quantitative and qualitative results.

翻译：以往的音频生成主要集中在指定声音类别上，如语音或音乐，其形式和内容受到很大限制。本文通过使用自然语言描述作为线索来生成广泛的声音，从而超越了特定音频生成的范畴。与视觉信息不同，文本描述本身简洁但蕴含丰富的隐藏含义，这为待生成的音频带来了更高的可能性和复杂性。我们采用变分量化的生成对抗网络（Variation-Quantized GAN）来训练一个码本，学习频谱图的离散表示。对于给定的文本描述，其预训练嵌入被输入到Transformer中，以采样码本索引，解码生成频谱图，再通过melgan声码器进一步转换为波形。生成的波形质量高、保真度强，且与给定的文本完美对应。实验表明，我们提出的方法能够生成自然、生动的音频，在定量和定性结果上均表现出色。