Deep neural networks have recently achieved breakthroughs in sound generation. Despite the outstanding sample quality, current sound generation models face issues on small-scale datasets (e.g., overfitting and low coverage of sound classes), significantly limiting performance. In this paper, we make the first attempt to investigate the benefits of pre-training on sound generation with AudioLDM, the cutting-edge model for audio generation, as the backbone. Our study demonstrates the advantages of the pre-trained AudioLDM, especially in data-scarcity scenarios. In addition, the baselines and evaluation protocol for sound generation systems are not consistent enough to compare different studies directly. Aiming to facilitate further study on sound generation tasks, we benchmark the sound generation task on various frequently-used datasets. We hope our results on transfer learning and benchmarks can provide references for further research on conditional sound generation.

翻译：深度神经网络近期在声音生成领域取得了突破性进展。尽管样本质量出色，但当前声音生成模型在小规模数据集上面临诸多问题（例如过拟合和声音类别覆盖不足），这严重限制了其性能。本文首次尝试以音频生成前沿模型AudioLDM为骨干网络，探究预训练对声音生成的益处。研究表明，预训练AudioLDM在数据稀缺场景下具有显著优势。此外，由于声音生成系统的基线和评估协议缺乏一致性，不同研究之间难以直接比较。为促进声音生成任务的进一步研究，我们在多个常用数据集上建立了声音生成任务的基准。希望本文关于迁移学习和基准测试的结果能为条件声音生成的后续研究提供参考。