Reconstructions of visual perception from brain activity have improved tremendously, but the practical utility of such methods has been limited. This is because such models are trained independently per subject where each subject requires dozens of hours of expensive fMRI training data to attain high-quality results. The present work showcases high-quality reconstructions using only 1 hour of fMRI training data. We pretrain our model across 7 subjects and then fine-tune on minimal data from a new subject. Our novel functional alignment procedure linearly maps all brain data to a shared-subject latent space, followed by a shared non-linear mapping to CLIP image space. We then map from CLIP space to pixel space by fine-tuning Stable Diffusion XL to accept CLIP latents as inputs instead of text. This approach improves out-of-subject generalization with limited training data and also attains state-of-the-art image retrieval and reconstruction metrics compared to single-subject approaches. MindEye2 demonstrates how accurate reconstructions of perception are possible from a single visit to the MRI facility. All code is available on GitHub.

翻译：视觉感知的脑活动重建技术已取得显著进步，但其实际应用价值仍有限。这是由于此类模型需针对每位被试独立训练，每个被试需耗费数十小时昂贵的fMRI训练数据才能获得高质量结果。本研究展示了仅使用1小时fMRI训练数据即可实现高质量重建。我们基于7名被试数据预训练模型，随后针对新被试的少量数据进行微调。本研究所提出的新型功能对齐方法，通过线性映射将所有脑数据对齐至共享被试潜在空间，再经由共享非线性映射进入CLIP图像空间。最后通过微调Stable Diffusion XL使其以CLIP潜在向量替代文本作为输入，完成从CLIP空间到像素空间的映射。该方法在有限训练数据下显著提升跨被试泛化能力，并在图像检索与重建指标上超越单被试方法。MindEye2证明仅需单次MRI扫描即可实现精确的感知重建。所有代码已在GitHub开源。