This paper presents BELT, a novel model and learning framework for the pivotal topic of brain-to-language translation research. The translation from noninvasive brain signals into readable natural language has the potential to promote the application scenario as well as the development of brain-computer interfaces (BCI) as a whole. The critical problem in brain signal decoding or brain-to-language translation is the acquisition of semantically appropriate and discriminative EEG representation from a dataset of limited scale and quality. The proposed BELT method is a generic and efficient framework that bootstraps EEG representation learning using off-the-shelf large-scale pretrained language models (LMs). With a large LM's capacity for understanding semantic information and zero-shot generalization, BELT utilizes large LMs trained on Internet-scale datasets to bring significant improvements to the understanding of EEG signals. In particular, the BELT model is composed of a deep conformer encoder and a vector quantization encoder. Semantical EEG representation is achieved by a contrastive learning step that provides natural language supervision. We achieve state-of-the-art results on two featuring brain decoding tasks including the brain-to-language translation and zero-shot sentiment classification. Specifically, our model surpasses the baseline model on both tasks by 5.45% and over 10% and archives a 42.31% BLEU-1 score and 67.32% precision on the main evaluation metrics for translation and zero-shot sentiment classification respectively.

翻译：本文提出BELT——一种面向脑-语言翻译研究关键课题的新型模型与学习框架。将非侵入性脑信号转化为可读自然语言的技术，有望推动脑机接口（BCI）的应用场景拓展及整体发展。脑信号解码或脑-语言翻译的核心问题在于，如何从规模有限、质量受限的数据集中获取语义恰当且具有判别性的脑电图（EEG）表征。所提出的BELT方法是一种通用高效框架，通过利用现成的大规模预训练语言模型（LM）来引导EEG表征学习。凭借大型语言模型理解语义信息与零样本泛化的能力，BELT利用在互联网规模数据集上训练的大语言模型显著提升了对EEG信号的理解水平。具体而言，BELT模型由深度卷积编码器与向量量化编码器组成，通过提供自然语言监督的对比学习步骤实现语义化的EEG表征。我们在两项具有代表性的脑解码任务——脑-语言翻译与零样本情感分类中取得了最先进结果。其中，本模型在两项任务上分别超越基线模型5.45%和超过10%，在翻译与零样本情感分类的核心评估指标上分别达到42.31%的BLEU-1得分与67.32%的精确率。