Towards an End-to-End Framework for Invasive Brain Signal Decoding with Large Language Models

In this paper, we introduce a groundbreaking end-to-end (E2E) framework fordecoding invasive brain signals, marking a significant advancement in the fieldof speech neuroprosthesis. Our methodology leverages the comprehensivereasoning abilities of large language models (LLMs) to facilitate directdecoding. By fully integrating LLMs, we achieve results comparable to thestate-of-the-art cascade models. Our findings underscore the immense potentialof E2E frameworks in speech neuroprosthesis, particularly as the technologybehind brain-computer interfaces (BCIs) and the availability of relevantdatasets continue to evolve. This work not only showcases the efficacy ofcombining LLMs with E2E decoding for enhancing speech neuroprosthesis but alsosets a new direction for future research in BCI applications, underscoring theimpact of LLMs in decoding complex neural signals for communicationrestoration. Code will be made available athttps://github.com/FsFrancis15/BrainLLM.