Foley sound generation aims to synthesise the background sound for multimedia content. Previous models usually employ a large development set with labels as input (e.g., single numbers or one-hot vector). In this work, we propose a diffusion model based system for Foley sound generation with text conditions. To alleviate the data scarcity issue, our model is initially pre-trained with large-scale datasets and fine-tuned to this task via transfer learning using the contrastive language-audio pertaining (CLAP) technique. We have observed that the feature embedding extracted by the text encoder can significantly affect the performance of the generation model. Hence, we introduce a trainable layer after the encoder to improve the text embedding produced by the encoder. In addition, we further refine the generated waveform by generating multiple candidate audio clips simultaneously and selecting the best one, which is determined in terms of the similarity score between the embedding of the candidate clips and the embedding of the target text label. Using the proposed method, our system ranks ${1}^{st}$ among the systems submitted to DCASE Challenge 2023 Task 7. The results of the ablation studies illustrate that the proposed techniques significantly improve sound generation performance. The codes for implementing the proposed system are available online.

翻译：拟音生成旨在为多媒体内容合成背景声音。以往模型通常使用带有标签（例如单个数字或独热向量）的大型开发集作为输入。在这项工作中，我们提出了一种基于扩散模型的系统，用于在文本条件下进行拟音生成。为了缓解数据稀缺问题，我们的模型首先使用大规模数据集进行预训练，然后通过基于对比语言-音频预训练（CLAP）技术的迁移学习微调至该任务。我们观察到，文本编码器提取的特征嵌入会显著影响生成模型的性能。因此，我们在编码器之后引入了一个可训练层，以改进编码器产生的文本嵌入。此外，我们通过同时生成多个候选音频片段并选择最佳片段来进一步优化生成的波形，该最佳片段根据候选片段嵌入与目标文本标签嵌入之间的相似度分数确定。使用所提出的方法，我们的系统在DCASE Challenge 2023 Task 7提交的系统中排名第一。消融研究结果表明，所提出的技术显著提升了声音生成性能。实现所提出系统的代码已在线公开。