Text-to-music generation models are now capable of generating high-quality music audio in broad styles. However, text control is primarily suitable for the manipulation of global musical attributes like genre, mood, and tempo, and is less suitable for precise control over time-varying attributes such as the positions of beats in time or the changing dynamics of the music. We propose Music ControlNet, a diffusion-based music generation model that offers multiple precise, time-varying controls over generated audio. To imbue text-to-music models with time-varying control, we propose an approach analogous to pixel-wise control of the image-domain ControlNet method. Specifically, we extract controls from training audio yielding paired data, and fine-tune a diffusion-based conditional generative model over audio spectrograms given melody, dynamics, and rhythm controls. While the image-domain Uni-ControlNet method already allows generation with any subset of controls, we devise a new strategy to allow creators to input controls that are only partially specified in time. We evaluate both on controls extracted from audio and controls we expect creators to provide, demonstrating that we can generate realistic music that corresponds to control inputs in both settings. While few comparable music generation models exist, we benchmark against MusicGen, a recent model that accepts text and melody input, and show that our model generates music that is 49% more faithful to input melodies despite having 35x fewer parameters, training on 11x less data, and enabling two additional forms of time-varying control. Sound examples can be found at https://MusicControlNet.github.io/web/.

翻译：文本到音乐生成模型现已能够生成高质量、风格广泛的音乐音频。然而，文本控制主要适用于操作全局音乐属性（如风格、情绪和速度），难以精确控制随时间变化的属性（如节拍的时间位置或音乐动态的变化）。我们提出Music ControlNet，一种基于扩散的音乐生成模型，可对生成的音频提供多个精确的时变控制。为了赋予文本到音乐模型时变控制能力，我们提出了一种类似于图像域ControlNet方法中像素级控制的策略。具体而言，我们从训练音频中提取控制信号并生成配对数据，在此基础上对基于扩散的条件生成模型进行微调，使其能够根据旋律、动态和节奏控制信号处理音频频谱图。尽管图像域的Uni-ControlNet方法已支持任意控制子集的生成，但我们设计了一种新策略，允许创作者输入仅在时间上部分指定的控制信号。我们分别对从音频中提取的控制信号以及创作者可能提供的控制信号进行了评估，结果表明，在这两种场景下，模型均能生成与输入控制信号相符的真实音乐。尽管目前可比的音乐生成模型较少，我们仍以近期支持文本和旋律输入的MusicGen模型为基准进行对比，展示出我们的模型在参数数量减少35倍、训练数据量减少11倍且具备两种额外时变控制形式的情况下，生成音乐对输入旋律的忠实度提升了49%。音频示例请访问 https://MusicControlNet.github.io/web/。