We propose Diffusion Inference-Time T-Optimization (DITTO), a general-purpose frame-work for controlling pre-trained text-to-music diffusion models at inference-time via optimizing initial noise latents. Our method can be used to optimize through any differentiable feature matching loss to achieve a target (stylized) output and leverages gradient checkpointing for memory efficiency. We demonstrate a surprisingly wide-range of applications for music generation including inpainting, outpainting, and looping as well as intensity, melody, and musical structure control - all without ever fine-tuning the underlying model. When we compare our approach against related training, guidance, and optimization-based methods, we find DITTO achieves state-of-the-art performance on nearly all tasks, including outperforming comparable approaches on controllability, audio quality, and computational efficiency, thus opening the door for high-quality, flexible, training-free control of diffusion models. Sound examples can be found at https://DITTO-Music.github.io/web/.

翻译：我们提出扩散推理时间T优化（DITTO），这是一种通用框架，通过在推理时优化初始噪声潜变量来控制预训练的文本到音乐扩散模型。该方法可通过任意可微特征匹配损失进行优化，以实现目标（风格化）输出，并利用梯度检查点技术提升内存效率。我们展示了该方法在音乐生成领域的广泛适用性，包括补全、外延、循环生成以及强度、旋律和音乐结构控制——所有这些均未对底层模型进行微调。将我们的方法与相关训练方法、引导方法和基于优化的方法对比后，发现DITTO在几乎所有任务上均达到最先进水平，包括在可控性、音频质量和计算效率方面优于同类方法，从而为扩散模型的高质量、灵活、免训练控制开辟了新途径。音频示例可在https://DITTO-Music.github.io/web/ 获取。