State-of-the-art systems neuroscience experiments yield large-scale multimodal data, and these data sets require new tools for analysis. Inspired by the success of large pretrained models in vision and language domains, we reframe the analysis of large-scale, cellular-resolution neuronal spiking data into an autoregressive spatiotemporal generation problem. Neuroformer is a multimodal, multitask generative pretrained transformer (GPT) model that is specifically designed to handle the intricacies of data in systems neuroscience. It scales linearly with feature size, can process an arbitrary number of modalities, and is adaptable to downstream tasks, such as predicting behavior. We first trained Neuroformer on simulated datasets, and found that it both accurately predicted simulated neuronal circuit activity, and also intrinsically inferred the underlying neural circuit connectivity, including direction. When pretrained to decode neural responses, the model predicted the behavior of a mouse with only few-shot fine-tuning, suggesting that the model begins learning how to do so directly from the neural representations themselves, without any explicit supervision. We used an ablation study to show that joint training on neuronal responses and behavior boosted performance, highlighting the model's ability to associate behavioral and neural representations in an unsupervised manner. These findings show that Neuroformer can analyze neural datasets and their emergent properties, informing the development of models and hypotheses associated with the brain.

翻译：最先进的系统神经科学实验产生大规模多模态数据，这些数据集需要新的分析工具。受视觉和语言领域大规模预训练模型成功经验的启发，我们将大规模细胞分辨率神经元锋电位数据的分析重构为一个自回归时空生成问题。Neuroformer是一种专门为处理系统神经科学数据复杂性而设计的多模态多任务生成式预训练Transformer（GPT）模型。该模型随特征规模呈线性扩展，可处理任意数量的模态，并适用于下游任务（如行为预测）。我们首先在模拟数据集上训练Neuroformer，发现其不仅能准确预测模拟神经回路活动，还能内在推断包括方向性在内的潜在神经回路连接。当模型通过预训练解码神经响应后，仅需少量样本微调即可预测小鼠行为，这表明模型可直接从神经表征本身开始学习如何执行该任务，无需任何显式监督。通过消融实验我们证明，联合训练神经响应与行为可提升模型性能，凸显了模型以无监督方式关联行为表征与神经表征的能力。这些发现表明，Neuroformer能够分析神经数据集及其涌现特性，为大脑相关模型与假说的开发提供依据。