Foley sound generation aims to synthesise the background sound for multimedia content. Previous models usually employ a large development set with labels as input (e.g., single numbers or one-hot vector). In this work, we propose a diffusion model based system for Foley sound generation with text conditions. To alleviate the data scarcity issue, our model is initially pre-trained with large-scale datasets and fine-tuned to this task via transfer learning using the contrastive language-audio pertaining (CLAP) technique. We have observed that the feature embedding extracted by the text encoder can significantly affect the performance of the generation model. Hence, we introduce a trainable layer after the encoder to improve the text embedding produced by the encoder. In addition, we further refine the generated waveform by generating multiple candidate audio clips simultaneously and selecting the best one, which is determined in terms of the similarity score between the embedding of the candidate clips and the embedding of the target text label. Using the proposed method, our system ranks ${1}^{st}$ among the systems submitted to DCASE Challenge 2023 Task 7. The results of the ablation studies illustrate that the proposed techniques significantly improve sound generation performance. The codes for implementing the proposed system are available online.

翻译：福利用户生成声音旨在为多媒体内容合成背景音。先前模型通常使用带有标签（如单数字或独热向量）的大型开发集作为输入。本文提出一种基于扩散模型的系统，用于在文本条件下生成福利用户声音。为缓解数据稀缺问题，我们的模型首先在大规模数据集上预训练，并通过对比语言-音频预训练（CLAP）技术，利用迁移学习微调至该任务。我们观察到，文本编码器提取的特征嵌入会显著影响生成模型的性能。因此，我们在编码器后引入一个可训练层，以改善编码器生成的文本嵌入。此外，我们通过同时生成多个候选音频片段，并根据候选片段嵌入与目标文本标签嵌入之间的相似度得分选择最佳片段，进一步优化生成波形。采用所提出的方法，我们的系统在DCASE 2023挑战赛任务7提交的系统中排名第一。消融研究结果表明，所提出的技术显著提升了声音生成性能。实现该系统的代码已在线公开。