Foley sound generation aims to synthesise the background sound for multimedia content. Previous models usually employ a large development set with labels as input (e.g., single numbers or one-hot vector). In this work, we propose a diffusion model based system for Foley sound generation with text conditions. To alleviate the data scarcity issue, our model is initially pre-trained with large-scale datasets and fine-tuned to this task via transfer learning using the contrastive language-audio pertaining (CLAP) technique. We have observed that the feature embedding extracted by the text encoder can significantly affect the performance of the generation model. Hence, we introduce a trainable layer after the encoder to improve the text embedding produced by the encoder. In addition, we further refine the generated waveform by generating multiple candidate audio clips simultaneously and selecting the best one, which is determined in terms of the similarity score between the embedding of the candidate clips and the embedding of the target text label. Using the proposed method, our system ranks ${1}^{st}$ among the systems submitted to DCASE Challenge 2023 Task 7. The results of the ablation studies illustrate that the proposed techniques significantly improve sound generation performance. The codes for implementing the proposed system are available online.

翻译：拟声生成旨在为多媒体内容合成背景音效。现有模型通常使用包含标签（如单一数字或独热向量）的大型开发集作为输入。本研究提出一种基于扩散模型的文本条件拟声生成系统。为解决数据稀缺问题，我们的模型首先在大规模数据集上进行预训练，然后通过基于对比语言-音频预训练（CLAP）技术的迁移学习微调至该任务。我们观察到文本编码器提取的特征嵌入会显著影响生成模型的性能，因此在编码器后引入可训练层以改进文本嵌入质量。此外，我们通过同时生成多个候选音频片段并依据候选片段嵌入与目标文本标签嵌入的相似度得分选择最优结果，进一步优化生成波形。采用所提方法，我们的系统在DCASE 2023挑战赛第七任务中位列提交系统第一名。消融实验结果证明，所提技术显著提升了声音生成性能。该系统实现代码已在线公开。