by Scientists at New York University (NYU) have made significant strides in understanding the complex neural processes that underlie speech production. In a groundbreaking study, the researchers used advanced neural networks to recreate speech from brain recordings and analyze the mechanisms involved in human speech.
Human speech production is a multifaceted behavior that involves both motor control and sensory processing. Disentangling the intricate processes of feedback and feedforward during speech production has posed a challenge for scientists. However, by applying innovative deep learning techniques to human neurosurgical recordings, the NYU researchers were able to decode speech parameters from cortical signals.
Using neural network architectures that distinguish between causal, anticausal, and noncausal temporal convolutions, the researchers meticulously analyzed the contributions of feedforward and feedback in speech production. This approach allowed them to unravel the processing of neural signals that occur simultaneously during speech production and the recognition of one’s own voice.
The findings of the study challenge existing notions that separate feedback and feedforward cortical networks. The researchers discovered a nuanced architecture of mixed feedback and feedforward processing across frontal and temporal cortices. This newfound perspective, combined with the remarkable speech decoding performance of their neural networks, represents a significant step forward in understanding the complex neural mechanisms underlying speech production.
The researchers have also leveraged this new perspective to develop prostheses that can read brain activity and translate it directly into speech. What sets their prototype apart from others in the field is its ability to recreate the patient’s voice using only a small dataset of recordings. By incorporating a deep neural network that considers a latent auditory space and can be trained on a few samples of an individual voice, such as a YouTube video or Zoom recording, the researchers can provide patients with their own voice after losing the ability to speak.
To collect the necessary data, the researchers worked with a group of patients with refractory epilepsy, a condition that is currently untreatable with medication. These patients had a grid of subdural EEG electrodes implanted on their brains for a week-long monitoring period. In addition, they provided researchers with valuable insights into brain activity during speech production.
The team of researchers from NYU involved in this study includes Adeen Flinker, Associate Professor of Biomedical Engineering at NYU Tandon and Neurology at the NYU Grossman School of Medicine; Yao Wang, Professor of Biomedical Engineering and Electrical and Computer Engineering at NYU Tandon and a member of NYU WIRELESS; Ran Wang, Xupeng Chen, and Amirhossein Khalilian-Gourtani from NYU Tandon’s Electrical and Computer Engineering Department; Leyao Yu from the Biomedical Engineering Department; Patricia Dugan, Daniel Friedman, and Orrin Devinsky from NYU Grossman’s Neurology Department; and Werner Doyle from the Neurosurgery Department.
This groundbreaking research not only advances our understanding of speech production but also holds tremendous potential for individuals who have lost their ability to speak. The ability to recreate a patient’s own voice using neural networks opens up new possibilities for speech-producing prostheses and offers a ray of hope for those affected by speech disorders.
*Note:
- Source: Coherent Market Insights, Public sources, Desk research
- We have leveraged AI tools to mine information and compile it
