Foley sound presents the background sound for multimedia content and the generation of Foley sound involves computationally modelling sound effects with specialized techniques. In this work, we proposed a system for DCASE 2023 challenge task 7: Foley Sound Synthesis. The proposed system is based on AudioLDM, which is a diffusion-based text-to-audio generation model. To alleviate the data-hungry problem, the system first trained with large-scale datasets and then downstreamed into this DCASE task via transfer learning. Through experiments, we found out that the feature extracted by the encoder can significantly affect the performance of the generation model. Hence, we improve the results by leveraging the input label with related text embedding features obtained by a significant language model, i.e., contrastive language-audio pertaining (CLAP). In addition, we utilize a filtering strategy to further refine the output, i.e. by selecting the best results from the candidate clips generated in terms of the similarity score between the sound and target labels. The overall system achieves a Frechet audio distance (FAD) score of 4.765 on average among all seven different classes, substantially outperforming the baseline system which performs a FAD score of 9.7.

翻译：拟音声音为多媒体内容提供背景音效，其生成涉及利用专门技术对声音效果进行计算建模。本文针对DCASE 2023挑战赛任务7“拟音声音合成”提出了一套系统。该系统基于AudioLDM——一种基于扩散的文本到音频生成模型。为缓解数据稀缺问题，系统首先在大规模数据集上训练，随后通过迁移学习将其适配至本DCASE任务。实验发现，编码器提取的特征会显著影响生成模型的性能。因此，我们通过利用输入标签与经显著语言模型（即对比语言-音频预训练模型CLAP）获取的相关文本嵌入特征来改进结果。此外，我们采用过滤策略进一步优化输出，即根据声音与目标标签之间的相似度得分，从生成的候选片段中选取最优结果。该整体系统在全部七个不同类别上的弗雷歇音频距离（FAD）平均得分为4.765，大幅优于基线系统的9.7分。