We propose Diffusion Inference-Time T-Optimization (DITTO), a general-purpose frame-work for controlling pre-trained text-to-music diffusion models at inference-time via optimizing initial noise latents. Our method can be used to optimize through any differentiable feature matching loss to achieve a target (stylized) output and leverages gradient checkpointing for memory efficiency. We demonstrate a surprisingly wide-range of applications for music generation including inpainting, outpainting, and looping as well as intensity, melody, and musical structure control - all without ever fine-tuning the underlying model. When we compare our approach against related training, guidance, and optimization-based methods, we find DITTO achieves state-of-the-art performance on nearly all tasks, including outperforming comparable approaches on controllability, audio quality, and computational efficiency, thus opening the door for high-quality, flexible, training-free control of diffusion models. Sound examples can be found at https://DITTO-Music.github.io/web/.

翻译：我们提出了扩散推理时间T优化（DITTO），这是一个通用框架，用于在推理时通过优化初始噪声潜在变量来控制预训练的文本到音乐扩散模型。我们的方法可通过任何可微分的特征匹配损失进行优化，以实现目标（风格化）输出，并利用梯度检查点技术以提高内存效率。我们展示了该方法在音乐生成中令人惊讶的广泛应用，包括修复、扩展、循环，以及强度、旋律和音乐结构控制——所有这些都无需对基础模型进行微调。当我们将我们的方法与相关的基于训练、引导和优化的方法进行比较时，发现DITTO在几乎所有任务上都达到了最先进的性能，包括在可控性、音频质量和计算效率方面优于同类方法，从而为扩散模型实现高质量、灵活且无需训练的控制打开了大门。音频示例可在 https://DITTO-Music.github.io/web/ 找到。