Audio editing is applicable for various purposes, such as adding background sound effects, replacing a musical instrument, and repairing damaged audio. Recently, some diffusion-based methods achieved zero-shot audio editing by using a diffusion and denoising process conditioned on the text description of the output audio. However, these methods still have some problems: 1) they have not been trained on editing tasks and cannot ensure good editing effects; 2) they can erroneously modify audio segments that do not require editing; 3) they need a complete description of the output audio, which is not always available or necessary in practical scenarios. In this work, we propose AUDIT, an instruction-guided audio editing model based on latent diffusion models. Specifically, AUDIT has three main design features: 1) we construct triplet training data (instruction, input audio, output audio) for different audio editing tasks and train a diffusion model using instruction and input (to be edited) audio as conditions and generating output (edited) audio; 2) it can automatically learn to only modify segments that need to be edited by comparing the difference between the input and output audio; 3) it only needs edit instructions instead of full target audio descriptions as text input. AUDIT achieves state-of-the-art results in both objective and subjective metrics for several audio editing tasks (e.g., adding, dropping, replacement, inpainting, super-resolution). Demo samples are available at https://audit-demo.github.io/.

翻译：音频编辑可应用于多种场景，例如添加背景音效、更换乐器以及修复受损音频。近年来，一些基于扩散的方法通过利用扩散与去噪过程，以输出音频的文本描述作为条件，实现了零样本音频编辑。然而，这些方法仍存在以下问题：1）它们未针对编辑任务进行训练，难以保证良好的编辑效果；2）它们可能会错误地修改无需编辑的音频片段；3）它们需要输出音频的完整描述，而这在实际场景中并非总可获得或必要。为了解决这些问题，本文提出AUDIT——一种基于潜在扩散模型的指令引导音频编辑模型。具体而言，AUDIT具有三大设计特色：1）我们针对不同的音频编辑任务构建三元组训练数据（指令、输入音频、输出音频），并以指令和待编辑输入音频为条件训练扩散模型，生成编辑后的输出音频；2）通过对比输入与输出音频的差异，模型能够自动学习仅修改需要编辑的片段；3）文本输入仅需编辑指令，而非完整的目标音频描述。在多项音频编辑任务（例如添加、删除、替换、修复、超分辨率）中，AUDIT在客观指标和主观指标上均取得了最先进的性能。演示样本请访问 https://audit-demo.github.io/。