Researchers develop deep-learning method for translating vocal signals from the brain to text

In a nutshell: People who have lost the ability to speak due to severe paralysis might have hope for a better way to communicate, thanks to technology that essentially reads their thoughts and translates them to words and complete sentences in real-time. By tapping into the part of the brain that controls the voice box, researchers have figured out how to read those signals and turn them into words with sophisticated AI.

Researchers at the University of Califonia San Francisco developed a way to allow people with speech loss to communicate with their brains. The technology uses neural networks to translate brainwaves into words and phrases. It is a breakthrough because until now, the best neuroprosthetic technology has provided is letter-by-letter translations, which is very slow.

Aside from the highly refined algorithms, the new method taps the part of the brain that sends signals to the voicebox. Previous techniques used brain areas that controlled the hand or arm. It seems almost too obvious that scientists should have used this section of the brain from the start, but such is neurological research.

In developing the system, UCSF researchers recorded brain signals of volunteer subjects with unimpaired speech. The scientists fed the patterns to neural networks, which learned to decode them in real-time. They also applied a statistical language model to improve algorithm accuracy.

The researchers' main obstacle was whether or not the area that controls speech would function the same way in impaired subjects as it did with those with normal speech. So they employed an anonymous impaired volunteer (Bravo1) and worked to create a simple 50-word vocabulary to feed the algorithms---practical words he could use in everyday life like water, good, I, yes, no, and so forth.

During testing, the team asked Bravo1 simple questions such as "How are you?" or "Do you need anything?" When Bravo1 tried to reply verbally, the computer would translate his impulses into phrases like, "I am fine," or "No, I don't need anything."

"To our knowledge, this is the first successful demonstration of direct decoding of full words from the brain activity of someone who is paralyzed and cannot speak," said UCSF neurosurgeon Edward Chang and Professor Jeanne Robertson, senior author on the study. "It shows strong promise to restore communication by tapping into the brain's natural speech machinery."

The system is still relatively slow. It can translate up to 18 words a minute---for comparison, unimpaired people can speak at about 200 words per minute. Still, it is faster than any other previous neuroprosthetic system developed. It has a peak translation accuracy of 93 percent and a median of 75 percent.

However, this is just the start of an expanding study. The team's next steps are to increase the number of test subjects, expand the system's vocabulary, and improve the rate of speech translation.