Image reconstruction and captioning from brain activity evoked by visual stimuli allow researchers to further understand the connection between the human brain and the visual perception system. While deep generative models have recently been employed in this field, reconstructing realistic captions and images with both low-level details and high semantic fidelity is still a challenging problem. In this work, we propose UniBrain: Unify Image Reconstruction and Captioning All in One Diffusion Model from Human Brain Activity. For the first time, we unify image reconstruction and captioning from visual-evoked functional magnetic resonance imaging (fMRI) through a latent diffusion model termed Versatile Diffusion. Specifically, we transform fMRI voxels into text and image latent for low-level information and guide the backward diffusion process through fMRI-based image and text conditions derived from CLIP to generate realistic captions and images. UniBrain outperforms current methods both qualitatively and quantitatively in terms of image reconstruction and reports image captioning results for the first time on the Natural Scenes Dataset (NSD) dataset. Moreover, the ablation experiments and functional region-of-interest (ROI) analysis further exhibit the superiority of UniBrain and provide comprehensive insight for visual-evoked brain decoding.

翻译：视觉刺激诱发的脑活动进行图像重建与描述，使研究者能够进一步理解人脑与视觉感知系统之间的关联。尽管深度生成模型近期已被应用于该领域，但生成兼具低级细节与高级语义保真度的真实感描述和图像仍是极具挑战性的问题。本研究提出UniBrain：统一人类脑活动图像重建与描述的全合一扩散模型。我们首次通过名为Versatile Diffusion的潜在扩散模型，统一了基于视觉诱发功能磁共振成像（fMRI）的图像重建与描述。具体而言，我们将fMRI体素转化为文本和图像潜变量以获取低级信息，并通过基于fMRI的CLIP图像和文本条件引导反向扩散过程，生成真实描述与图像。UniBrain在图像重建方面，定性和定量结果均优于现有方法，并首次在自然场景数据集（NSD）上报告了图像描述结果。此外，消融实验和功能感兴趣区域（ROI）分析进一步展示了UniBrain的优越性，并为视觉诱发脑解码提供了全面见解。