In this paper, we introduce a groundbreaking end-to-end (E2E) framework for decoding invasive brain signals, marking a significant advancement in the field of speech neuroprosthesis. Our methodology leverages the comprehensive reasoning abilities of large language models (LLMs) to facilitate direct decoding. By fully integrating LLMs, we achieve results comparable to the state-of-the-art cascade models. Our findings underscore the immense potential of E2E frameworks in speech neuroprosthesis, particularly as the technology behind brain-computer interfaces (BCIs) and the availability of relevant datasets continue to evolve. This work not only showcases the efficacy of combining LLMs with E2E decoding for enhancing speech neuroprosthesis but also sets a new direction for future research in BCI applications, underscoring the impact of LLMs in decoding complex neural signals for communication restoration. Code will be made available at https://github.com/FsFrancis15/BrainLLM.

翻译：本文提出了一种开创性的端到端框架，用于解码侵入式脑信号，标志着言语神经假体领域的重大进展。我们的方法利用大语言模型强大的推理能力来实现直接解码。通过将LLM完全整合到框架中，我们取得了与当前最先进的级联模型相当的结果。我们的研究凸显了端到端框架在言语神经假体中的巨大潜力，尤其是在脑机接口技术及相关数据集持续发展的背景下。这项工作不仅展示了将LLM与端到端解码相结合以提升言语神经假体性能的有效性，也为未来BCI应用研究指明了新方向，突显了LLM在解码复杂神经信号以实现交流功能恢复方面的重要作用。代码将在https://github.com/FsFrancis15/BrainLLM 公开。