Dance-driven music generation aims to generate musical pieces conditioned on dance videos. Previous works focus on monophonic or raw audio generation, while the multiinstruments scenario is under-explored. The challenges of the dance-driven multi-instruments music (MIDI) generation are two-fold: 1) no publicly available multi-instruments MIDI and video paired dataset and 2) the weak correlation between music and video. To tackle these challenges, we build the first multi-instruments MIDI and dance paired dataset (D2MIDI). Based on our proposed dataset, we introduce a multi-instruments MIDI generation framework (Dance2MIDI) conditioned on dance video. Specifically, 1) to model the correlation between music and dance, we encode the dance motion using the GCN, and 2) to generate harmonious and coherent music, we employ Transformer to decode the MIDI sequence. We evaluate the generated music of our framework trained on D2MIDI dataset and demonstrate that our method outperforms existing methods. The data and code are available on the GitHub website.

翻译：舞蹈驱动的音乐生成旨在根据舞蹈视频生成音乐片段。以往的研究主要关注单声道或原始音频生成，而多乐器场景尚未得到充分探索。舞蹈驱动的多乐器音乐（MIDI）生成面临两个挑战：1）缺乏公开可用的多乐器MIDI与视频配对数据集；2）音乐与视频之间的弱相关性。为解决这些问题，我们构建了首个多乐器MIDI与舞蹈配对数据集（D2MIDI）。基于所提出的数据集，我们引入了一个以舞蹈视频为条件的多乐器MIDI生成框架（Dance2MIDI）。具体而言：1）为建模音乐与舞蹈之间的相关性，我们使用GCN对舞蹈动作进行编码；2）为生成和谐连贯的音乐，我们采用Transformer对MIDI序列进行解码。我们评估了在D2MIDI数据集上训练的框架生成的音乐，结果表明我们的方法优于现有方法。数据和代码已在GitHub网站上公开。