Audio editing is applicable for various purposes, such as adding background sound effects, replacing a musical instrument, and repairing damaged audio. Recently, some diffusion-based methods achieved zero-shot audio editing by using a diffusion and denoising process conditioned on the text description of the output audio. However, these methods still have some problems: 1) they have not been trained on editing tasks and cannot ensure good editing effects; 2) they can erroneously modify audio segments that do not require editing; 3) they need a complete description of the output audio, which is not always available or necessary in practical scenarios. In this work, we propose AUDIT, an instruction-guided audio editing model based on latent diffusion models. Specifically, AUDIT has three main design features: 1) we construct triplet training data (instruction, input audio, output audio) for different audio editing tasks and train a diffusion model using instruction and input (to be edited) audio as conditions and generating output (edited) audio; 2) it can automatically learn to only modify segments that need to be edited by comparing the difference between the input and output audio; 3) it only needs edit instructions instead of full target audio descriptions as text input. AUDIT achieves state-of-the-art results in both objective and subjective metrics for several audio editing tasks (e.g., adding, dropping, replacement, inpainting, super-resolution). Demo samples are available at https://audit-demo.github.io/.

翻译：音频编辑可用于多种场景，例如添加背景音效、替换乐器以及修复损坏音频。近期，部分基于扩散模型的方法通过将输出音频的文本描述作为条件，利用扩散与去噪过程实现了零样本音频编辑。然而，这些方法仍存在以下问题：1）未针对编辑任务进行训练，难以保证良好的编辑效果；2）可能错误修改无需编辑的音频片段；3）需要输出音频的完整描述，但在实际场景中这并非总是可行或必要。本文提出AUDIT，一种基于隐式扩散模型的指令引导音频编辑模型。具体而言，AUDIT包含三项核心设计：1）针对不同音频编辑任务构建三元组训练数据（指令、输入音频、输出音频），并以指令和待编辑输入音频为条件训练扩散模型，生成编辑后输出音频；2）通过比较输入与输出音频的差异，自动学习仅修改需编辑的片段；3）仅需编辑指令作为文本输入，无需目标音频的完整描述。在多项音频编辑任务（如添加、删除、替换、修复、超分辨率）中，AUDIT在客观指标与主观评价上均达到最优水平。示例音频请访问 https://audit-demo.github.io/。