The process of reconstructing experiences from human brain activity offers a unique lens into how the brain interprets and represents the world. In this paper, we introduce a method for reconstructing music from brain activity, captured using functional magnetic resonance imaging (fMRI). Our approach uses either music retrieval or the MusicLM music generation model conditioned on embeddings derived from fMRI data. The generated music resembles the musical stimuli that human subjects experienced, with respect to semantic properties like genre, instrumentation, and mood. We investigate the relationship between different components of MusicLM and brain activity through a voxel-wise encoding modeling analysis. Furthermore, we discuss which brain regions represent information derived from purely textual descriptions of music stimuli. We provide supplementary material including examples of the reconstructed music at https://google-research.github.io/seanet/brain2music

翻译：从人脑活动中重建体验的过程，为我们理解大脑如何诠释与表征世界提供了独特视角。本文提出了一种基于功能性磁共振成像（fMRI）数据重建音乐的方法。我们的方法通过两种路径实现：音乐检索或基于fMRI嵌入条件化的MusicLM音乐生成模型。生成的音乐在语义属性（如流派、配器、情绪）上与受试者体验的音乐刺激高度相似。我们通过体素级编码建模分析，探讨了MusicLM不同组件与脑活动之间的关系。此外，我们进一步分析了哪些脑区负责处理纯文本描述性音乐信息的表征。补充材料（含重建音乐示例）请参见https://google-research.github.io/seanet/brain2music