How to decode human vision through neural signals has attracted a long-standing interest in neuroscience and machine learning. Modern contrastive learning and generative models improved the performance of fMRI-based visual decoding and reconstruction. However, the high cost and low temporal resolution of fMRI limit their applications in brain-computer interfaces (BCIs), prompting a high need for EEG-based visual reconstruction. In this study, we present an EEG-based visual reconstruction framework. It consists of a plug-and-play EEG encoder called the Adaptive Thinking Mapper (ATM), which is aligned with image embeddings, and a two-stage EEG guidance image generator that first transforms EEG features into image priors and then reconstructs the visual stimuli with a pre-trained image generator. Our approach allows EEG embeddings to achieve superior performance in image classification and retrieval tasks. Our two-stage image generation strategy vividly reconstructs images seen by humans. Furthermore, we analyzed the impact of signals from different time windows and brain regions on decoding and reconstruction. The versatility of our framework is demonstrated in the magnetoencephalogram (MEG) data modality. We report that EEG-based visual decoding achieves SOTA performance, highlighting the portability, low cost, and high temporal resolution of EEG, enabling a wide range of BCI applications. The code of ATM is available at https://github.com/dongyangli-del/EEG_Image_decode.

翻译：如何通过神经信号解码人类视觉一直是神经科学与机器学习领域的长期研究热点。现代对比学习与生成模型已显著提升了基于fMRI的视觉解码与重建性能，但fMRI高昂的成本与较低的时间分辨率限制了其在脑机接口（BCI）中的应用，因此亟需发展基于脑电图（EEG）的视觉重建技术。本研究提出了一种基于EEG的视觉重建框架，包含即插即用的EEG编码器——自适应思维映射器（ATM），该编码器与图像嵌入对齐；以及两阶段EEG引导图像生成器，首先将EEG特征转化为图像先验，再通过预训练图像生成器重建视觉刺激。我们的方法使EEG嵌入在图像分类与检索任务中达到优越性能。两阶段图像生成策略生动重建了人类观察到的图像。此外，我们分析了不同时间窗口与脑区信号对解码与重建的影响，并在脑磁图（MEG）数据模态中验证了框架的通用性。实验表明，基于EEG的视觉解码达到当前最优（SOTA）性能，凸显了EEG的便携性、低成本与高时间分辨率优势，可支持广泛的BCI应用。ATM代码已开源：https://github.com/dongyangli-del/EEG_Image_decode。