The successful reconstruction of perceptual experiences from human brain activity has provided insights into the neural representations of sensory experiences. However, reconstructing arbitrary sounds has been avoided due to the complexity of temporal sequences in sounds and the limited resolution of neuroimaging modalities. To overcome these challenges, leveraging the hierarchical nature of brain auditory processing could provide a path toward reconstructing arbitrary sounds. Previous studies have indicated a hierarchical homology between the human auditory system and deep neural network (DNN) models. Furthermore, advancements in audio-generative models enable to transform compressed representations back into high-resolution sounds. In this study, we introduce a novel sound reconstruction method that combines brain decoding of auditory features with an audio-generative model. Using fMRI responses to natural sounds, we found that the hierarchical sound features of a DNN model could be better decoded than spectrotemporal features. We then reconstructed the sound using an audio transformer that disentangled compressed temporal information in the decoded DNN features. Our method shows unconstrained sounds reconstruction capturing sound perceptual contents and quality and generalizability by reconstructing sound categories not included in the training dataset. Reconstructions from different auditory regions remain similar to actual sounds, highlighting the distributed nature of auditory representations. To see whether the reconstructions mirrored actual subjective perceptual experiences, we performed an experiment involving selective auditory attention to one of overlapping sounds. The results tended to resemble the attended sound than the unattended. These findings demonstrate that our proposed model provides a means to externalize experienced auditory contents from human brain activity.

翻译：从人类大脑活动中成功重建感知体验，为理解感官体验的神经表征提供了新见解。然而，由于声音时间序列的复杂性以及神经影像模态分辨率的限制，任意声音的重建工作一直未能开展。为克服这些挑战，利用大脑听觉处理的层级特性可为实现任意声音重建提供路径。既往研究表明，人类听觉系统与深度神经网络（DNN）模型之间存在层级同源性。此外，音频生成模型的进步使得将压缩表征还原为高分辨率声音成为可能。本研究提出一种将听觉特征脑解码与音频生成模型相结合的新型声音重建方法。通过自然声音的fMRI响应，我们发现DNN模型的层级声音特征比频谱时间特征更易解码。继而利用音频变换器对解码后的DNN特征中分离的压缩时间信息进行声音重建。该方法实现了无约束声音重建，可捕捉声音的感知内容与质量，并通过重建训练数据集未包含的声音类别展现出泛化能力。不同听觉区域的重建结果均与实际声音保持相似性，凸显了听觉表征的分布式特性。为验证重建声音是否反映真实主观感知体验，我们开展了重叠声音中选择性听觉注意实验。结果显示，重建结果更倾向于接近被注意声音而非非注意声音。这些发现表明，本模型为从人类大脑活动中外化听觉体验内容提供了可行途径。