Our ability to use deep learning approaches to decipher neural activity would likely benefit from greater scale, in terms of both model size and datasets. However, the integration of many neural recordings into one unified model is challenging, as each recording contains the activity of different neurons from different individual animals. In this paper, we introduce a training framework and architecture designed to model the population dynamics of neural activity across diverse, large-scale neural recordings. Our method first tokenizes individual spikes within the dataset to build an efficient representation of neural events that captures the fine temporal structure of neural activity. We then employ cross-attention and a PerceiverIO backbone to further construct a latent tokenization of neural population activities. Utilizing this architecture and training framework, we construct a large-scale multi-session model trained on large datasets from seven nonhuman primates, spanning over 158 different sessions of recording from over 27,373 neural units and over 100 hours of recordings. In a number of different tasks, we demonstrate that our pretrained model can be rapidly adapted to new, unseen sessions with unspecified neuron correspondence, enabling few-shot performance with minimal labels. This work presents a powerful new approach for building deep learning tools to analyze neural data and stakes out a clear path to training at scale.

翻译：我们利用深度学习方法解读神经活动的能力，很可能受益于模型规模和数据集方面的更大尺度。然而，将多个神经记录整合到一个统一模型中颇具挑战，因为每次记录来自不同个体的不同神经元活动。本文提出了一种训练框架和架构，旨在对跨多样大规模神经记录的神经活动群体动态进行建模。我们的方法首先对数据集内的单个尖峰信号进行标记化，构建高效的神经事件表示，以捕捉神经活动的精细时间结构。随后，我们采用交叉注意力机制和PerceiverIO骨干网络，进一步构建神经群体活动的潜在标记化。利用这一架构和训练框架，我们构建了一个大规模多会话模型，该模型使用来自七只非人灵长类动物的大规模数据集进行训练，涵盖超过158个不同记录会话、超过27,373个神经单元以及100多小时的记录。在多个不同任务中，我们证明了预训练模型能够快速适应新的、未见过的会话，且无需指定神经元对应关系，从而在极少标签条件下实现小样本学习性能。本研究为构建用于分析神经数据的深度学习工具提供了一种强大的新方法，并开辟了一条向规模化训练的清晰路径。