Foley sound generation aims to synthesise the background sound for multimedia content. Previous models usually employ a large development set with labels as input (e.g., single numbers or one-hot vector). In this work, we propose a diffusion model based system for Foley sound generation with text conditions. To alleviate the data scarcity issue, our model is initially pre-trained with large-scale datasets and fine-tuned to this task via transfer learning using the contrastive language-audio pertaining (CLAP) technique. We have observed that the feature embedding extracted by the text encoder can significantly affect the performance of the generation model. Hence, we introduce a trainable layer after the encoder to improve the text embedding produced by the encoder. In addition, we further refine the generated waveform by generating multiple candidate audio clips simultaneously and selecting the best one, which is determined in terms of the similarity score between the embedding of the candidate clips and the embedding of the target text label. Using the proposed method, our system ranks ${1}^{st}$ among the systems submitted to DCASE Challenge 2023 Task 7. The results of the ablation studies illustrate that the proposed techniques significantly improve sound generation performance. The codes for implementing the proposed system are available online.

翻译：拟音生成旨在为多媒体内容合成背景声音。以往模型通常使用带有标签（例如单个数字或独热向量）的大规模开发集作为输入。在这项工作中，我们提出了一种基于扩散模型的文本条件拟音生成系统。为解决数据稀缺问题，我们的模型首先在大规模数据集上进行预训练，并通过对比语言-音频预训练（CLAP）技术进行迁移学习，微调至该任务。我们观察到文本编码器提取的特征嵌入会显著影响生成模型的性能。因此，我们在编码器后引入一个可训练层，以改进编码器生成的文本嵌入。此外，我们通过同时生成多个候选音频片段，并根据候选片段嵌入与目标文本标签嵌入之间的相似度分数选择最佳片段，进一步优化生成的波形。采用所提方法，我们的系统在DCASE Challenge 2023任务7中提交的系统里排名第一。消融实验结果表明，所提技术显著提升了声音生成性能。实现所提系统的代码已在线发布。