Brain signals translated into speech using AI(nature.com)
nature.com
Brain signals translated into speech using AI
https://www.nature.com/articles/d41586-019-01328-x
86 comments
I saw the article in the NYT. It mentions that the system is capable of 150 wpm, compared to older systems that could only do 8 wpm.
“Previous implant-based communication systems have produced about eight words a minute. The new program generates about 150 a minute, the pace of natural speech.”
https://www.nytimes.com/2019/04/24/health/artificial-speech-...
“Previous implant-based communication systems have produced about eight words a minute. The new program generates about 150 a minute, the pace of natural speech.”
https://www.nytimes.com/2019/04/24/health/artificial-speech-...
Thanks—since the paper is paywalled and HN tends to prefer the best popular article on such a topic, we've changed the URL to that from https://www.nature.com/articles/s41586-019-1119-1.
A pdf of the paper itself is linked from elsewhere in this thread.
A pdf of the paper itself is linked from elsewhere in this thread.
Nature has an editorial on the paper that is better and free
https://www.nature.com/articles/d41586-019-01328-x
https://www.nature.com/articles/d41586-019-01328-x
Ok, we've switched to that from https://www.nytimes.com/2019/04/24/health/artificial-speech-.... Thanks!
Sadly the comment section on nytimes.com is absolutely appalling...
I usually find the NYTimes comments mostly fine, save for the expected ideological bias.
I read your comment, went back to check comments, and couldn't believe my eyes. People think this is mind-reading.
> the article is using a benevolence trope ... to cloak the real purpose of this research — interfaces for perfectly healthy (and likely rich) people to control maybe their car or television with their minds (which would be cool!).
> Soon employers will be able to require that all employees will have their thoughts recorded while performing any work-related task, as "performance of these tasks carries no expectations of privacy."
So many people missing the fact that the technique (ECoG) requires a surgical procedure. It's not a consumer product, and it won't be cheap or easy any time soon.
Is there a Gell-Mann amnesia effect [1] for comments sections?
[1] https://en.wikipedia.org/wiki/Gell-Mann_amnesia_effect
I read your comment, went back to check comments, and couldn't believe my eyes. People think this is mind-reading.
> the article is using a benevolence trope ... to cloak the real purpose of this research — interfaces for perfectly healthy (and likely rich) people to control maybe their car or television with their minds (which would be cool!).
> Soon employers will be able to require that all employees will have their thoughts recorded while performing any work-related task, as "performance of these tasks carries no expectations of privacy."
So many people missing the fact that the technique (ECoG) requires a surgical procedure. It's not a consumer product, and it won't be cheap or easy any time soon.
Is there a Gell-Mann amnesia effect [1] for comments sections?
[1] https://en.wikipedia.org/wiki/Gell-Mann_amnesia_effect
While your points are far more relevant, I feel obligated to point out also, that no employer would wish to hear the thoughts of their employees, because it would not be at all pleasant for them to hear all of that. Imagine working with employees who say out loud exactly what they are thinking all the time. No one wants that, not even (especially?) control freaks.
[deleted]
I'm not an expert in this field, but it seems like any mind reading we can do today with ECoG we can do tomorrow (or next decade) with EEG; the invasive electrodes make it easier, but we should be able to read signals non-invasively with better clarity in the future. When you say it won't be cheap/easy "any time soon," what does that mean to you? 10 years? 50? 100? The privacy concerns seem valid to me, though I'm not saying the sky is falling. I look forward to thinking text into a computer. I'd be extremely skeptical of any company demanding that I put their hardware on my scalp, but I'd be willing to put my hardware on my scalp and present some limited interface between it and the company hardware. Also, I expect the problems with invasive surgery to decline.
I would be moderately surprised if you could infer ECOG at 150 wpm via EEG. That would be roughly akin to claiming that, because you can infer water flow in an pipeline to Los Angeles from magnetometers welded to the pipe, you can do the same thing using a constellation of geosynchronous satellites.
There's a lot of things happened in between the pipe and outer space. Similarly, there's a lot of things happening between the arachnoid mater and the surface of the skin: innumerable liquid-solid interfaces of varying types being a major class of issues. The electric activity in the arrector pili, the galea, etc. Distance, bone, etc.
There's a lot of things happened in between the pipe and outer space. Similarly, there's a lot of things happening between the arachnoid mater and the surface of the skin: innumerable liquid-solid interfaces of varying types being a major class of issues. The electric activity in the arrector pili, the galea, etc. Distance, bone, etc.
I didn't say "everything," this is kind of a side-tangent but in particular I imagine writing to the brain might be really difficult to do non-invasively while keeping per-write side-effects as low as invasive alternatives. I disagree with your projections of the future, and expect that geosynchronous satellites will be able to infer water-flow in a given (non-shielded) pipe within the next 30 years. I could be wrong.
To be fair those ideas about mind-reading aren't absurd to bring up. Yes this device requires ECoG, but maybe this paves the road to figure out how to do something similar with an EEG helmet.
The employer listening to your thoughts thing? yea I doubt it. Maybe possible at some point but unlikely it would be normalized anytime soon.
The employer listening to your thoughts thing? yea I doubt it. Maybe possible at some point but unlikely it would be normalized anytime soon.
Nutty, but not too far off from predictions I see on HN from time to time. Just replace mind reading with facial recognition or voice analysis...
+1 For introducing me to the Gell-Mann amnesia effect which I didn't know had a name before
I'd love to see if it works when people are dreaming. Or people who have Locked-in Syndrome. Or even people who are in a coma.
https://en.wikipedia.org/wiki/Locked-in_syndrome
https://en.wikipedia.org/wiki/Locked-in_syndrome
It doesn't try to read thoughts. It uses commands that are sent to the tongue/mouth to form syllables. We're nowhere near being able to trace / decode thoughts.
Actually you can, in limited cases.
If you have prior knowledge of the person's brain activity when looking at an image, it's possible to predict (reconstruct an image of) what they're looking at.
https://journals.plos.org/ploscompbiol/article/figures?id=10...
If you have prior knowledge of the person's brain activity when looking at an image, it's possible to predict (reconstruct an image of) what they're looking at.
https://journals.plos.org/ploscompbiol/article/figures?id=10...
That is not reading thoughts. That is via the occipital Cortex which has a highly structured representation of what is projected on to the retina. This is much easier as the organisation of the data is maintained relative to the retina, just regrouped according to visual field rather than eye.
If we're reading data from the brain isn't that by definition reading thoughts, regardless of where in the brain they originate from?
Part of the study involved decoding images from memory/recall. I did say _limited_.
It depends how tightly you want to define "thought", but being able to think of an object and having a computer display a reasonable representation of it seems like a good start.
It depends how tightly you want to define "thought", but being able to think of an object and having a computer display a reasonable representation of it seems like a good start.
10 years ago I watched a national geographic documentary about something quite similar to this one. The results were also apperantly impressive. It's weird they haven't progressed much during this period. Sadly I can't find any resources about that project anywhere.
Probably because of this little important detailed not mentioned in the title "Five participants underwent chronic implantation of a high-density, subdural electrode array over
the lateral surface of the brain as part of their clinical treatment for epilepsy"
Or people with conditions similar to Hawkins. I know one person with a rare (unknown) ailment who can only mumble sounds.. could make his existence .. well it's hard to think about it
I haven't read the full paper to know what hardware it needs, but if it can be put into a headphone like wearable and integrated into phone to be output as text, then we can type on phone as fast as we can think. Looking forward to it! Also once of the major impediments to using voice assistants in public is to not want to shout into the phone and if it can be directly integrated into the assistant, the voice assistant use will explode. The possibilities are amazing!
This technology uses an electrode array referred to as ECoG, which needs to be surgically placed on the surface of the brain. Current technology does not make it possible to have the signal quality from non-invasive methods.
Ok. Then all those applications will have to wait!
Sounds like a neural network that converts EEG signals to ECoG signals would be an interesting idea.
If there is enough information in EEG signals to generate speech, an intervening layer to "convert" EEG signals into ECoG signals is an unnecessary complication.
Unfortunately, there probably isn't; on top of that, EEGs are extremely susceptible to noise from nerve signals going to the head/face, such as clenching your jaw or raising your eyebrows
ECoG has much higher temporal and spatial resolution than EEG. Conversion from EEG to ECoG is not likely, the opposite can be done, but not very useful though.
[deleted]
Why are advances like this always presented as useful in a medical context?
It seems to me like some sort of fear of talking about commercial use cases - for example in this case the ability to talk in your mind with your phone, person-phone telepathy if you wish
It seems to me like some sort of fear of talking about commercial use cases - for example in this case the ability to talk in your mind with your phone, person-phone telepathy if you wish
Because a slight increase in convenience to be able to talk to your phone subvocally is trivial compared to granting capability for vocal communication to someone who is incapable of regular speech. One is a slight improvement in convenience for an otherwise able person. The other is life-changing.
True, but many people weigh the convenience by the number of uses.
In that case, 0.0001 * 1 million >> 1.0 * 100
In that case, 0.0001 * 1 million >> 1.0 * 100
I think Star Trek II put it well, in that the needs of the many are not always more important than the needs of the few.
I think you mean Star Trek III: The Search for Spock? :)
You're right - I had mis-remembered it. The scene I was thinking of was the one near the end of Wrath of Khan where Spock and Kirk are discussing sacrifice, but that's not where the quote is from.
I'll blame it on the scene in II being so much more memorable. ;)
I'll blame it on the scene in II being so much more memorable. ;)
Are there really that many people willing to have electrodes implanted in their brain just to... type on their phone!?
If this was consumer-grade technology, you wouldn't need to type on your phone, you could simply communicate with others directly like telepathy.
Not right now, but imagine if the level of invasiveness/danger was on the same level as a birth control implant, or botox, or lasik. I've had surgery to remove drainage tubes from my eardrums just because they were being inconvenient. I'd rather put up with that in exchange for being able to do 150wpm hands-free.
What are you going to do with your hands while you are concentrating on your writing? They are going to be dangling there, useless for nothing except reflex actions like picking your nose or driving your car (!).
I already have attached devices that let me enter text faster than I can think it. I think the tech only gets interesting for the handy-capable when it goes beyond words, capturing images or sounds as you imagine them, and that is a long way of.
I already have attached devices that let me enter text faster than I can think it. I think the tech only gets interesting for the handy-capable when it goes beyond words, capturing images or sounds as you imagine them, and that is a long way of.
I might be holding my phone, I might just not be near a keyboard, I might be controlling a video game and use it for extra input...
And while I can type sentences pretty fast, sometimes I get bogged down with things like code and entering special characters and being able to subvocalize a single syllable to activate macros would be pretty nice. (Yes I know you could do that without a brain interface, but the point is that hands are limited.)
Edit: why does this comment get a downvote?
And while I can type sentences pretty fast, sometimes I get bogged down with things like code and entering special characters and being able to subvocalize a single syllable to activate macros would be pretty nice. (Yes I know you could do that without a brain interface, but the point is that hands are limited.)
Edit: why does this comment get a downvote?
These use cases are all great for speech recognition, which is decades ahead of brain interfaces and less invasive (and just as quiet if you want by sub-vocalizing into a throat mike). Again, not particularly interesting to most of us but very interesting to people with various disabilities.
Honestly, for a few thousand Euros, if it worked really well (and was a well supported product software wise, with open APIs) ... I might do it - And I do think that while it wouldn't be everyone, significantly many people willing to do it could be found.
Only on Hacker News could people's main concern about whether to have invasive surgery to put electrodes on the surface of the speech centres of the brain be ensuring the APIs of the software it transmits messages to are open!
I'm looking forward to the day that a technology like this is safe or simple enough for widespread commercial use. I would be concerned about the safety of brain surgery, but lots of people get lasik which uses lasers to cut your eyes, which in my head is much scarier. Hopefully someday this could be a fairly routine procedure.
Selling subvocal control could fund the disability case. Finding users willing to pay to build the technology you want is very much in vogue.
It's where the money's at:
> This work was supported by grants from the NIH (DP2 OD008627 and U01 NS098971-01). E.F.C. is a New York Stem Cell Foundation-Robertson Investigator. This research was also supported by The William K. Bowes Foundation, the Howard Hughes Medical Institute, The New York Stem Cell Foundation and The Shurl and Kay Curci Foundation
(Acknowledgements section, p. 498.)
> This work was supported by grants from the NIH (DP2 OD008627 and U01 NS098971-01). E.F.C. is a New York Stem Cell Foundation-Robertson Investigator. This research was also supported by The William K. Bowes Foundation, the Howard Hughes Medical Institute, The New York Stem Cell Foundation and The Shurl and Kay Curci Foundation
(Acknowledgements section, p. 498.)
Another factor that no one has mentioned is that this approach seems to require invasive brain surgery. This is likely not a large risk for someone suffering from locked-in syndrome, but would be for someone who wants to improve their typing speed.
As such, most early use cases would be restricted to people with few alternatives, such as medical cases.
As such, most early use cases would be restricted to people with few alternatives, such as medical cases.
Per another comment [1]
"This technology uses an electrode array referred to as ECoG, which needs to be surgically placed on the surface of the brain. Current technology does not make it possible to have the signal quality from non-invasive methods."
[1] : https://news.ycombinator.com/item?id=19742479
"This technology uses an electrode array referred to as ECoG, which needs to be surgically placed on the surface of the brain. Current technology does not make it possible to have the signal quality from non-invasive methods."
[1] : https://news.ycombinator.com/item?id=19742479
I predict that you'll see this deployable by nanobots within a decade.
I'd bet you 100USD that isn't true.
Or with neural lace some time sooner...
"Musk defined the neuro lace as a "digital layer above the cortex" that would not necessarily imply extensive surgical insertion but ideally an implant through a vein or artery" well it is still pretty undefined. Essentially neural lace can be anything enabling this kind of communication.
As other people have already said, it's where the grant money is. That's why many of the top "statistics" programs in the United States are really biostatistics programs - They get more funding and research opportunities and all of the benefits that come with those.
I think the study of cortical implants is interesting for this question because they touch the same questions basically.
Cortical implants basically form the opposite side by providing a direct interface between machines and the brain. They are already available and deployed in medical contexts. Why aren't they deployed in healthy people as well?
They have pretty cool advantages. With them, you can listen to music, etc. without being affected by outside noises (unless you want to). Imagine having a phone call in a noisy environment and you can actually hear stuff. Imagine your neighbours having a loud party but you can still sleep tight because you turned it into sleep mode where only fire alarms may disturb you. How many conflicts you could avoid!
However, cortical implants have heavy disadvantages: They give you worse hearing quality than normal hearing, they require surgery, and also need devices on the outside that are in contact with them. That's why they aren't useful for healthy humans yet. But maybe, with improvement, one day they will become.
Basically, the problems of this technology will be the same: worse quality, requirement for surgery, requirement for extra devices to carry around.
Cortical implants basically form the opposite side by providing a direct interface between machines and the brain. They are already available and deployed in medical contexts. Why aren't they deployed in healthy people as well?
They have pretty cool advantages. With them, you can listen to music, etc. without being affected by outside noises (unless you want to). Imagine having a phone call in a noisy environment and you can actually hear stuff. Imagine your neighbours having a loud party but you can still sleep tight because you turned it into sleep mode where only fire alarms may disturb you. How many conflicts you could avoid!
However, cortical implants have heavy disadvantages: They give you worse hearing quality than normal hearing, they require surgery, and also need devices on the outside that are in contact with them. That's why they aren't useful for healthy humans yet. But maybe, with improvement, one day they will become.
Basically, the problems of this technology will be the same: worse quality, requirement for surgery, requirement for extra devices to carry around.
In many cases there are multiple avenues for application, but this paper says the research was specifically funded by the National Institute of Health, which means the PR focus will be toward human health/disease.
> Why are advances like this always presented as useful in a medical context?
You don't think that people who have lost the ability to speak, e.g. from stroke would find it useful?
You don't think that people who have lost the ability to speak, e.g. from stroke would find it useful?
I believe it is about grants. If your research has medical implications, there may be more grants available.
Because commercial use requiring brain surgery is most likely not going to work
you can't get funding for these sorts of things unless you show medical context first. It may change- billionaires funding brain scanning companies are an exmaple, but even then they tend to work with quadriplegics.
If you cure a quadriplegic, then getting approval for the general public tends to be easy (it's an easy sell) but the other way around... not so much.
If you cure a quadriplegic, then getting approval for the general public tends to be easy (it's an easy sell) but the other way around... not so much.
I don't fully understand.
> As each participant recited hundreds of sentences, the electrodes recorded the firing patterns of neurons in the motor cortex. The researchers associated those patterns with the subtle movements of the patient’s lips, tongue, larynx and jaw that occur during natural speech. The team then translated those movements into spoken sentences.
So this means the audio example is speech synthesized from data gemerated while the person was actually reading out loud, right?
Why does the text under the headline claim "no muscle movement needed", which would imply audible speech synthesized from mere thoughts?
> As each participant recited hundreds of sentences, the electrodes recorded the firing patterns of neurons in the motor cortex. The researchers associated those patterns with the subtle movements of the patient’s lips, tongue, larynx and jaw that occur during natural speech. The team then translated those movements into spoken sentences.
So this means the audio example is speech synthesized from data gemerated while the person was actually reading out loud, right?
Why does the text under the headline claim "no muscle movement needed", which would imply audible speech synthesized from mere thoughts?
Former neuroscientist working on similar stuff here.
While the flashy part is the “speech synthesis”, the science breakthrough is actually better frames as an machine learning problem.
Imagine you record someone moving their hand across a canvas. The hand movement becomes the input, the drawing the output. The ML problem they solved is to reconstruct the output (or at least something that resembles it) from the input. In the study’s case, that’s efferent motor signals.
There is a long history of mapping these kind of signals in the sensorimotor homunculus to the respective muscles they control downstream (and some really cool stuff like prosthetic limbs can already be controlled with it), but speaking is a notoriously hard motor task and requires a lot of muscles to work in unison in very precise ways. When you implant these multi-electrode arrays, you get a few hundred more or less random single neurons, astrocytes, and local field potentials from the nether in between. Nonsense, noisy data. Being able to map this back to the result they produce in the body is technically as complex as astonishing!
While the flashy part is the “speech synthesis”, the science breakthrough is actually better frames as an machine learning problem.
Imagine you record someone moving their hand across a canvas. The hand movement becomes the input, the drawing the output. The ML problem they solved is to reconstruct the output (or at least something that resembles it) from the input. In the study’s case, that’s efferent motor signals.
There is a long history of mapping these kind of signals in the sensorimotor homunculus to the respective muscles they control downstream (and some really cool stuff like prosthetic limbs can already be controlled with it), but speaking is a notoriously hard motor task and requires a lot of muscles to work in unison in very precise ways. When you implant these multi-electrode arrays, you get a few hundred more or less random single neurons, astrocytes, and local field potentials from the nether in between. Nonsense, noisy data. Being able to map this back to the result they produce in the body is technically as complex as astonishing!
There's a picture of the electrode array on the Nature site, and it looks about 3cm x 3cm, with about 16 x 16 sensors. So 256 inputs to the analysis system.
I know nothing about these kinds of electrodes, but how sensitive do they have to be? Sub-microvolt? And how fast? Sub-millisecond?
Finally, on the results themselves. If I read it right, the sensors had been previously implanted into epilepsy patients, and were "re-purposed" for this study.
So I assume they patients were able to speak? If so, it's trickier to demonstrate advantages for those who have lost the ability to speak.
I know nothing about these kinds of electrodes, but how sensitive do they have to be? Sub-microvolt? And how fast? Sub-millisecond?
Finally, on the results themselves. If I read it right, the sensors had been previously implanted into epilepsy patients, and were "re-purposed" for this study.
So I assume they patients were able to speak? If so, it's trickier to demonstrate advantages for those who have lost the ability to speak.
Participants spoke sentences while their neural activity was recorded. The acoustic speech was associated with motor movement which was in turn associated with learned neural patterns. As such the final result is a multi-stage model which can translate from patient specific (at a minimum in the sense of electrode configuration) neural activity into audio.
All neural activity (from what I saw in the paper) is neural activity while the individual is reading sentences out loud.
All neural activity (from what I saw in the paper) is neural activity while the individual is reading sentences out loud.
Yes, they are synthesizing speech from thought while the person mimicked speaking moving their mouth but making no sound.
> Although synthesis performance for mimed speech was inferior to the performance for audible speech—which is probably due to absence of phonation signals during miming—this demonstrates that it is possible to decode important spectral features of speech that were never audibly uttered (P < 1 × 10−11 compared to chance, n = 58; Wilcoxon signedrank test) and that the decoder did not rely on auditory feedback
> Although synthesis performance for mimed speech was inferior to the performance for audible speech—which is probably due to absence of phonation signals during miming—this demonstrates that it is possible to decode important spectral features of speech that were never audibly uttered (P < 1 × 10−11 compared to chance, n = 58; Wilcoxon signedrank test) and that the decoder did not rely on auditory feedback
Okay. I now understand what's actually significant about this - and conclude that "no muscle movement needed" is a lie.
If/when driverless cars are reliable, human's learning driving skills may become obsolete.
Wonder what are the implications if we are able to reliably communicate our thoughts using AI. Can training vocal cords to speak become an obsolete skill too? On this trajectory - I also wonder what other core capabilities can be delegated to AI and what role this could plays in evolution of future generations.
After working on the neural speech-prosthesis problem in the single (or nearly single) electrode case, this seems to be a promising result from the ECoG side of things. While there's a few more results I'd like to see about how things would be expected to generalize more broadly over time/neural-input, it does look like the results are fairly solid.
Hopefully there can be some further development from here into some practical applications.
Hopefully there can be some further development from here into some practical applications.
Can't wait to control vim using my brain some day
You could imagine Neuro link is doing something similar.
[deleted]
[deleted]
Does this require mouth movement though?
[deleted]
I read this as scientist creates free speech from brain signals.