Speech brain-computer interfaces (BCIs) aim to restore communication for people with paralysis by translating neural activity into text. Most systems use cascaded frameworks that decode phonemes before assembling sentences with an n-gram language model (LM), preventing joint optimization of all stages simultaneously. Here, we introduce an end-to-end Brain-to-Text (BIT) framework that translates neural activity into coherent sentences using a single differentiable neural network. Central to our approach is a cross-task, cross-species pretrained neural encoder, whose representations transfer to both attempted and imagined speech. In a cascaded setting with an n-gram LM, the pretrained encoder establishes a new state-of-the-art (SOTA) on the Brain-to-Text '24 and '25 benchmarks. Integrated end-to-end with audio large language models (LLMs) and trained with contrastive learning for cross-modal alignment, BIT reduces the word error rate (WER) of the prior end-to-end method from 24.69% to 10.22%. Notably, we find that small-scale audio LLMs markedly improve end-to-end decoding. Beyond record-setting performance, BIT aligns attempted and imagined speech embeddings to enable cross-task generalization. Altogether, our approach advances the integration of large, diverse neural datasets, paving the way for an end-to-end decoding framework that supports seamless, differentiable optimization.

翻译：语音脑机接口旨在通过将神经活动转化为文本，为瘫痪患者恢复沟通能力。现有系统多采用级联框架，先解码音素，再利用n-gram语言模型组合句子，阻碍了所有阶段的联合优化。本文提出端到端脑到文本框架，利用单一可微神经网络将神经活动转化为连贯语句。该框架的核心是跨任务、跨物种预训练神经编码器，其表征可迁移至尝试性语音和想象性语音。在采用n-gram语言模型的级联设置中，预训练编码器在Brain-to-Text '24和'25基准测试中达到新的最优性能。通过与音频大语言模型进行端到端集成，并采用对比学习训练实现跨模态对齐，BIT将此前端到端方法的词错误率从24.69%降至10.22%。值得注意的是，我们发现小规模音频LLM能显著改善端到端解码性能。除创纪录的性能外，BIT通过对齐尝试性语音和想象性语音嵌入实现跨任务泛化。总体而言，本方法推动了大规模多样化神经数据集的整合，为支持无缝可微优化的端到端解码框架奠定基础。