Visualizing the insights of the invisible music is able to bring listeners an enjoyable and immersive listening experience, and therefore has attracted much attention in the field of information visualization. Over the past decades, various music visualization techniques have been introduced. However, most of them are manually designed by following the visual encoding rules, thus shown in form of a graphical visual representation whose visual encoding schema is usually taking effort to understand. Recently, some researchers use figures or illustrations to represent music moods, lyrics, and musical features, which are more intuitive and attractive. However, in these techniques, the figures are usually pre-selected or statically generated, so they cannot precisely convey insights of different pieces of music. To address this issue, in this paper, we introduce MusicJam, a music visualization system that is able to generate narrative illustrations to represent the insight of the input music. The system leverages a novel generation model designed based on GPT-2 to generate meaningful lyrics given the input music and then employs the stable diffusion model to transform the lyrics into coherent illustrations. Finally, the generated results are synchronized and rendered as an MP4 video accompanied by the input music. We evaluated the proposed lyric generation model by comparing it to the baseline models and conducted a user study to estimate the quality of the generated illustrations and the final music videos. The results showed the power of our technique.

翻译：揭示无形音乐中的洞见能够为听众带来愉悦且沉浸的聆听体验，因此引起了信息可视化领域的广泛关注。过去数十年间，多种音乐可视化技术被相继提出。然而，这些技术大多遵循视觉编码规则进行人工设计，因此以图形化视觉表征的形式呈现，其视觉编码方案通常需要费解。近年来，部分研究者采用图形或插画来表现音乐情绪、歌词及音乐特征，这些方式更为直观且富有吸引力。但此类技术中，图像通常预先选定或静态生成，因而无法精确传达不同音乐作品的洞见。为解决这一问题，本文提出MusicJam——一种能够生成叙事插图以呈现输入音乐洞见的音乐可视化系统。该系统利用基于GPT-2设计的创新生成模型，根据输入音乐生成富有意义的歌词，随后采用稳定扩散模型将歌词转化为连贯的插画。最终，生成的图像与输入音乐同步并渲染为MP4视频。我们通过与基线模型对比评估了所提出的歌词生成模型，并开展用户研究检验生成插图及最终音乐视频的质量。结果表明了本技术的有效性。