Brain encoding models aim to predict brain voxel-wise responses to stimuli images, replicating brain signals captured by neuroimaging techniques. There is a large volume of publicly available data, but training a comprehensive brain encoding model is challenging. The main difficulties stem from a) diversity within individual brain, with functional heterogeneous brain regions; b) diversity of brains from different subjects, due to genetic and developmental differences; c) diversity of imaging modalities and processing pipelines. We use this diversity to our advantage by introducing the All-for-One training recipe, which divides the challenging one-big-model problem into multiple small models, with the small models aggregating the knowledge while preserving the distinction between the different functional regions. Agnostic of the training recipe, we use biological knowledge of the brain, specifically retinotopy, to introduce inductive bias to learn a 3D brain-to-image mapping that ensures a) each neuron knows which image regions and semantic levels to gather information, and b) no neurons are left behind in the model. We pre-trained a brain encoding model using over one million data points from five public datasets spanning three imaging modalities. To the best of our knowledge, this is the most comprehensive brain encoding model to the date. We demonstrate the effectiveness of the pre-trained model as a drop-in replacement for commonly used vision backbone models. Furthermore, we demonstrate the application of the model to brain decoding. Code and the model checkpoint will be made available.

翻译：脑编码模型旨在预测大脑对刺激图像的体素级响应，复现神经影像技术捕捉的脑部信号。尽管存在大量公开数据，训练一个全面的脑编码模型仍具挑战性。主要困难源于：a) 个体脑内功能异质性脑区的多样性；b) 遗传和发育差异导致的不同被试者脑部多样性；c) 成像模态和处理流程的多样性。我们利用这种多样性，通过引入"全能一体式"训练方案，将棘手的单一大型模型问题分解为多个小模型，这些小模型在保留不同功能区域区分性的同时整合知识。该方案与具体训练策略无关，我们利用大脑的生物知识（尤其是视网膜拓扑）引入归纳偏置，学习三维脑-图像映射，确保：a) 每个神经元知晓从哪些图像区域和语义层级获取信息，b) 模型中没有神经元被遗漏。我们使用跨越三种成像模态的五个公开数据集中超过一百万个数据点，预训练了一个脑编码模型。据我们所知，这是迄今为止最全面的脑编码模型。实验证明该预训练模型可作为常用视觉骨干模型的即插即用替代品。此外，我们还展示了该模型在脑解码中的应用。代码与模型检查点将开源。