Visualizing the insights of the invisible music is able to bring listeners an enjoyable and immersive listening experience, and therefore has attracted much attention in the field of information visualization. Over the past decades, various music visualization techniques have been introduced. However, most of them are manually designed by following the visual encoding rules, thus shown in form of a graphical visual representation whose visual encoding schema is usually taking effort to understand. Recently, some researchers use figures or illustrations to represent music moods, lyrics, and musical features, which are more intuitive and attractive. However, in these techniques, the figures are usually pre-selected or statically generated, so they cannot precisely convey insights of different pieces of music. To address this issue, in this paper, we introduce MusicJam, a music visualization system that is able to generate narrative illustrations to represent the insight of the input music. The system leverages a novel generation model designed based on GPT-2 to generate meaningful lyrics given the input music and then employs the stable diffusion model to transform the lyrics into coherent illustrations. Finally, the generated results are synchronized and rendered as an MP4 video accompanied by the input music. We evaluated the proposed lyric generation model by comparing it to the baseline models and conducted a user study to estimate the quality of the generated illustrations and the final music videos. The results showed the power of our technique.

翻译：可视化无形音乐中的洞察能够为听众带来愉悦且沉浸式的聆听体验，因此在信息可视化领域引起了广泛关注。过去几十年中，各种音乐可视化技术被提出。然而，大多数技术遵循视觉编码规则手动设计，以图形化视觉表示形式呈现，其视觉编码方案通常需要费解。近年来，一些研究者使用图形或插图来表示音乐情绪、歌词和音乐特征，这些方式更直观且更具吸引力。然而，在这些技术中，图形通常是预先选择或静态生成的，因此无法精确传达不同音乐作品的洞察。为解决这一问题，本文介绍了MusicJam，一个能够生成叙事插图以表示输入音乐洞察的音乐可视化系统。该系统利用基于GPT-2设计的新型生成模型，根据输入音乐生成有意义的歌词，然后采用稳定扩散模型将歌词转化为连贯的插图。最后，生成的结果与输入音乐同步并渲染为MP4视频。我们通过与基线模型对比评估了所提出的歌词生成模型，并开展了一项用户研究以评估生成插图及最终音乐视频的质量。结果证明了我们技术的有效性。