This paper presents a study on the use of a real-time music-to-image system as a mechanism to support and inspire musicians during their creative process. The system takes MIDI messages from a keyboard as input which are then interpreted and analysed using state-of-the-art generative AI models. Based on the perceived emotion and music structure, the system's interpretation is converted into visual imagery that is presented in real-time to musicians. We conducted a user study in which musicians improvised and composed using the system. Our findings show that most musicians found the generated images were a novel mechanism when playing, evidencing the potential of music-to-image systems to inspire and enhance their creative process.

翻译：本文研究了一种实时音乐到图像系统作为支持与激发音乐家创作过程的机制。该系统以键盘输入的MIDI信号作为输入，通过最先进的生成式AI模型进行解析与分析。基于感知到的情感与音乐结构，系统将解析结果转化为视觉图像并实时呈现给音乐家。我们开展了一项用户研究，让音乐家使用该系统进行即兴演奏与作曲。研究发现，大多数音乐家认为演奏时生成的图像是一种新颖的交互机制，这证明了音乐到图像系统在激发与增强创作过程方面的潜力。