Foley sound generation aims to synthesise the background sound for multimedia content. Previous models usually employ a large development set with labels as input (e.g., single numbers or one-hot vector). In this work, we propose a diffusion model based system for Foley sound generation with text conditions. To alleviate the data scarcity issue, our model is initially pre-trained with large-scale datasets and fine-tuned to this task via transfer learning using the contrastive language-audio pertaining (CLAP) technique. We have observed that the feature embedding extracted by the text encoder can significantly affect the performance of the generation model. Hence, we introduce a trainable layer after the encoder to improve the text embedding produced by the encoder. In addition, we further refine the generated waveform by generating multiple candidate audio clips simultaneously and selecting the best one, which is determined in terms of the similarity score between the embedding of the candidate clips and the embedding of the target text label. Using the proposed method, our system ranks ${1}^{st}$ among the systems submitted to DCASE Challenge 2023 Task 7. The results of the ablation studies illustrate that the proposed techniques significantly improve sound generation performance. The codes for implementing the proposed system are available online.

翻译：拟声生成旨在为多媒体内容合成背景音效。以往模型通常使用带有标签（例如单个数字或独热向量）的大型开发集作为输入。在本工作中，我们提出了一种基于扩散模型的系统，用于在文本条件下生成拟声。为缓解数据稀缺问题，我们的模型首先在大规模数据集上进行预训练，并通过对比语言-音频预训练（CLAP）技术进行迁移学习以微调至该任务。我们观察到，由文本编码器提取的特征嵌入会显著影响生成模型的性能。因此，我们在编码器之后引入了一个可训练层，以改进编码器产生的文本嵌入。此外，我们同时生成多个候选音频片段，并根据候选片段嵌入与目标文本标签嵌入之间的相似度得分选取最佳片段，从而进一步优化生成的波形。采用所提出的方法，我们的系统在DCASE挑战赛2023任务7中提交的系统中排名第一。消融研究的结果表明，所提出的技术显著提升了声音生成性能。实现所提出系统的代码已在线发布。