Foley sound generation aims to synthesise the background sound for multimedia content, which involves computationally modelling sound effects with specialized techniques. In this work, we proposed a diffusion based generative model for DCASE 2023 challenge task 7: Foley Sound Synthesis. The proposed system is based on AudioLDM, which is a diffusion-based text-to-audio generation model. To alleviate the data scarcity of the task 7 training set, our model is initially trained with large-scale datasets and downstream into this DCASE task via transfer learning. We have observed that the feature extracted by the encoder can significantly affect the performance of the generation model. Hence, we improve the results by leveraging the input label with related text embedding features obtained by a large language model, i.e., contrastive language-audio pretraining (CLAP). In addition, we utilize a filtering strategy to further refine the output, i.e. by selecting the best results from the candidate clips generated in terms of the similarity score between the sound and target labels. The overall system achieves a Frechet audio distance (FAD) score of 4.765 on average among all seven different classes, substantially outperforming the baseline system which achieves a FAD score of 9.7.

翻译：Foley声音生成旨在为多媒体内容合成背景音，涉及通过专门技术对音效进行计算建模。本研究针对DCASE 2023挑战赛任务七（Foley声音合成）提出了一种基于扩散的生成模型。该模型以AudioLDM为基础，后者是一种基于扩散的文本到音频生成模型。为缓解任务七训练集数据稀缺问题，我们的模型首先在大规模数据集上进行预训练，随后通过迁移学习适配至该DCASE任务。实验发现编码器提取的特征会显著影响生成模型的性能，因此我们通过利用大语言模型（即对比语言-音频预训练模型，CLAP）获得的输入标签相关文本嵌入特征来优化结果。此外，我们采用筛选策略进一步优化输出——基于声音与目标标签之间的相似度得分，从生成的候选音频片段中选取最佳结果。整体系统在全部七个不同类别上平均实现4.765的弗雷歇音频距离（FAD）得分，显著优于基线系统9.7的FAD得分。