We propose Polyffusion, a diffusion model that generates polyphonic music scores by regarding music as image-like piano roll representations. The model is capable of controllable music generation with two paradigms: internal control and external control. Internal control refers to the process in which users pre-define a part of the music and then let the model infill the rest, similar to the task of masked music generation (or music inpainting). External control conditions the model with external yet related information, such as chord, texture, or other features, via the cross-attention mechanism. We show that by using internal and external controls, Polyffusion unifies a wide range of music creation tasks, including melody generation given accompaniment, accompaniment generation given melody, arbitrary music segment inpainting, and music arrangement given chords or textures. Experimental results show that our model significantly outperforms existing Transformer and sampling-based baselines, and using pre-trained disentangled representations as external conditions yields more effective controls.

翻译：摘要：我们提出Polyffusion，这是一种将复调音乐视作类图像钢琴卷帘表示进行生成的扩散模型。该模型支持两种可控音乐生成范式：内部控制与外部控制。内部控制指用户预定义音乐片段后由模型补全剩余部分的过程，类似于掩码音乐生成（或音乐修补）任务；外部控制则通过交叉注意力机制，利用和弦、织体等外部关联信息对模型进行条件约束。实验证明，通过内部与外部控制的结合，Polyffusion统一了包括给定伴奏生成旋律、给定旋律生成伴奏、任意音乐片段修补、以及给定和弦或织体进行音乐编配在内的多种音乐创作任务。结果表明，我们的模型显著优于现有基于Transformer与采样的基线方法，且采用预训练解耦表示作为外部条件能实现更有效的控制。