How to decode human vision through neural signals has attracted a long-standing interest in neuroscience and machine learning. Modern contrastive learning and generative models improved the performance of fMRI-based visual decoding and reconstruction. However, the high cost and low temporal resolution of fMRI limit their applications in brain-computer interfaces (BCIs), prompting a high need for EEG-based visual reconstruction. In this study, we present an EEG-based visual reconstruction framework. It consists of a plug-and-play EEG encoder called the Adaptive Thinking Mapper (ATM), which is aligned with image embeddings, and a two-stage EEG guidance image generator that first transforms EEG features into image priors and then reconstructs the visual stimuli with a pre-trained image generator. Our approach allows EEG embeddings to achieve superior performance in image classification and retrieval tasks. Our two-stage image generation strategy vividly reconstructs images seen by humans. Furthermore, we analyzed the impact of signals from different time windows and brain regions on decoding and reconstruction. The versatility of our framework is demonstrated in the magnetoencephalogram (MEG) data modality. We report that EEG-based visual decoding achieves SOTA performance, highlighting the portability, low cost, and high temporal resolution of EEG, enabling a wide range of BCI applications. The code of ATM is available at https://github.com/dongyangli-del/EEG_Image_decode.

翻译：如何通过神经信号解码人类视觉一直是神经科学和机器学习领域长期关注的问题。现代对比学习与生成模型提升了基于fMRI的视觉解码与重建性能，但fMRI的高成本与低时间分辨率限制了其在脑机接口中的应用，因此亟需开发基于脑电的视觉重建方法。本研究提出了一种基于脑电的视觉重建框架，包含一个即插即用的脑电编码器——自适应思维映射器（ATM），该编码器与图像嵌入对齐；以及一个两阶段脑电引导图像生成器，首先将脑电特征转化为图像先验，再通过预训练图像生成器重建视觉刺激。我们的方法使脑电嵌入在图像分类与检索任务中达到卓越性能，两阶段图像生成策略生动地重建了人类看到的图像。此外，我们分析了不同时间窗口与脑区信号对解码与重建的影响，并在脑磁图数据模态中验证了框架的通用性。结果表明，基于脑电的视觉解码实现了当前最优性能，凸显了脑电的便携性、低成本与高时间分辨率优势，可支持广泛的脑机接口应用。ATM代码已开源：https://github.com/dongyangli-del/EEG_Image_decode。