Facebook drops funding for interface that reads the brain(technologyreview.com)
technologyreview.com
Facebook drops funding for interface that reads the brain
https://www.technologyreview.com/2021/07/14/1028447/facebook-brain-reading-interface-stops-funding/
68 comments
I would imagine that after about the 300th word, they'd start to lose all meaning and their associated signals would collapse much closer together. It definitely does feel like an extremely optimistic experiment. Do you think there's a good way to capture enough useful data in a lab environment, short neural implants + microphones with 24/7 recording?
>Do you think there's a good way to capture enough useful data in a lab environment, short neural implants + microphones with 24/7 recording?
It's a hard problem, even neutral implants and mics isn't enough. I had two months to come up with some ideas about how we could get a human to produce reliable signals, and failed completely. That was when I walked away and got a job.
It's a hard problem, even neutral implants and mics isn't enough. I had two months to come up with some ideas about how we could get a human to produce reliable signals, and failed completely. That was when I walked away and got a job.
I know nothing about neuroscience and sounded like a recipe for failure, you obviously have to use true communications, like asking the test subject between food choices and actually bringing the food requested, so the next time they are asked about food choices they know is a real interaction and have to make a real decision, the same for everything else that is tested, meaning make it as real as physically possible.
You are missing the point, they are not decoding semantics. The signals being decoded correspond to imagined motor activations needed to produce the word. It's similar to asking you to mentally rehearse throwing a ball.
I don't think it is anything like simulating a mechanical action like throwing a ball, just saying a word alters brains in completely different ways depending on the person, for example saying "spider" reminds me of spiderman and there is little I can to stop such through from happening, to someone else it may remind them of something completely different, even my own thoughts about that word may be different any other day.
To actually capture words you have much better chances reading the brain while writing the word with a pen because you are actually sending a signal from the brain to your hand, which is what they did here (even if he doesn't actually have a hand to move, the brain still can emit the very same commands): https://www.cnet.com/google-amp/news/brain-implants-let-para...
To actually capture words you have much better chances reading the brain while writing the word with a pen because you are actually sending a signal from the brain to your hand, which is what they did here (even if he doesn't actually have a hand to move, the brain still can emit the very same commands): https://www.cnet.com/google-amp/news/brain-implants-let-para...
Grandparent here, who studied the PhD.
>I don't think it is anything like simulating a mechanical action like throwing a ball, just saying a word alters brains in completely different ways depending on the person
It's exactly like imagining throwing a ball. A disproportionately large amount of the motor cortex is used for facial muscle and tongue control. Look up the "cortical homunculus".
>for example saying "spider" reminds me of spiderman and there is little I can to stop such through from happening
These BCIs can't tell whether or not you're thinking about Spider-Man, otherwise they'd be used on terrorists to get information. Instead, they're mainly focused on broad fitting synchronisation of M1 neurons (indicating rest)
>To actually capture words you have much better chances reading the brain while writing the word with a pen because you are actually sending a signal from the brain to your hand
Real movement does produce much more consistent results than imagined movements, but it doesn't translate well to the target market for BCIs (people with severe motor disabilities)
>even if he doesn't actually have a hand to move, the brain still can emit the very same commands
It doesn't work like that in real life. With no feedback, we're back to square one with the "imagined movement".
>I don't think it is anything like simulating a mechanical action like throwing a ball, just saying a word alters brains in completely different ways depending on the person
It's exactly like imagining throwing a ball. A disproportionately large amount of the motor cortex is used for facial muscle and tongue control. Look up the "cortical homunculus".
>for example saying "spider" reminds me of spiderman and there is little I can to stop such through from happening
These BCIs can't tell whether or not you're thinking about Spider-Man, otherwise they'd be used on terrorists to get information. Instead, they're mainly focused on broad fitting synchronisation of M1 neurons (indicating rest)
>To actually capture words you have much better chances reading the brain while writing the word with a pen because you are actually sending a signal from the brain to your hand
Real movement does produce much more consistent results than imagined movements, but it doesn't translate well to the target market for BCIs (people with severe motor disabilities)
>even if he doesn't actually have a hand to move, the brain still can emit the very same commands
It doesn't work like that in real life. With no feedback, we're back to square one with the "imagined movement".
Fascinating, thanks for the explanation.
Sure, a word can evoke all sorts of meaning, other brain process, etc. The decoding principle leveraged here is exactly as I indicated. It's decoding motor activation associated with imagining speaking a word, from the speech-motor cortex.
The study you quoted uses similar principles.
The study you quoted uses similar principles.
Given the diversity of pronunciation even within one local group, and that everyone learns to make the sounds their own mechanical way, how does this generalize between individuals?
Maybe our thoughts aren't reducible to material reality?
Maybe just like everything in the history of mankind it is a material reality and we just dont understand it yet, just like we didn't understood how to fly for millions of years, maybe it will take a few more to read a brain.
> Maybe just like everything in the history of mankind it is a material reality
There are at least a gazillion things in the history of mankind that we have not been able to reduce to a material reality.
There are at least a gazillion things in the history of mankind that we have not been able to reduce to a material reality.
Just like people used to believe the sun was a god or something magical for centuries but now we know it's a mass of gas under nuclear fusion. In general believing that something we don't understand happens due magic (or however you wanna call non-material reality) has a bad track record.
Use the phrase 'god' or 'something magical' as a synonym for 'something beyond our understanding and control' and you'll realize we're a lot closer to that level of understanding than we are to having figured everything out.
Abstraction makes it much easier to reason about complex systems, such as the world :)
Abstraction makes it much easier to reason about complex systems, such as the world :)
Don't you have knowledge of immaterial realities, like the lines, points, and angles of geometry? How is that possible if all you've got to think with is matter?
The machinations of the brain are as immaterial as the bytes of a hard drive; which technically speaking are both material given that the information there is an electrical property of matter.
On the contrary - the more we focus on technology and treat everything as if it's a machine, the less we understand the true nature of the human mind. That's why the general trend in literature, art and architecture is down instead of up.
Misleading title, it’s more of a delay according to Facebook’s quote in the article:
“While we still believe in the long-term potential of head-mounted optical [brain-computer interface] technologies, we’ve decided to focus our immediate efforts on a different neural interface approach that has a nearer-term path to market,” the company said.
“While we still believe in the long-term potential of head-mounted optical [brain-computer interface] technologies, we’ve decided to focus our immediate efforts on a different neural interface approach that has a nearer-term path to market,” the company said.
I don’t think it’s misleading, they have removed funding and suspended work.
Could they start again in the future? Sure, but in practical terms removing funding and stopping people from working on a project indefinitely is the same as cancelling. This is just corporate-speak, which requires translation - if removing funding, staff, research space and equipment and stopping all work indefinitely isn’t cancelling something, what extra steps are required for cancelling something?
The different neural approach is reading signals from muscles in your arms, so it’s definitely not the brain-speech interface they were talking about, but the fact they say it’s closer to market will placate investors.
Could they start again in the future? Sure, but in practical terms removing funding and stopping people from working on a project indefinitely is the same as cancelling. This is just corporate-speak, which requires translation - if removing funding, staff, research space and equipment and stopping all work indefinitely isn’t cancelling something, what extra steps are required for cancelling something?
The different neural approach is reading signals from muscles in your arms, so it’s definitely not the brain-speech interface they were talking about, but the fact they say it’s closer to market will placate investors.
I was on the list for a SDK at CTRL-Labs before FB swooped in and bought them. I'm really bummed I couldn't get ahold of one first.
In having on-again, off-again RSI issues over the years from coding, I think this tech would be hugely beneficial and is more realistic to ship in the near future.
In having on-again, off-again RSI issues over the years from coding, I think this tech would be hugely beneficial and is more realistic to ship in the near future.
I also have RSI and it stops me from being a productive programmer, playing any sort of video game, and completely ruined my life for a couple years. If I get really good sleep I can use the computer for a couple hours without any pain at all but normally I'm relegated to only using my hands for a little bit each day otherwise I'll injure them. I tried using voice to text but then I injured my vocal chords using it. Physical therapy and Rehabilitation is slow and I doubt I'll ever be able to have the full use of my voice and hands again. Having a BCI would improve my quality of life immensely.
Yeah, I was in a similar place at one point. Check out the book in this thread: https://news.ycombinator.com/item?id=6099623. The book is: https://www.amazon.com/Mindbody-Prescription-Healing-Body-Pa...
I don't know if it was the book, a placebo effect, or something else, but I got better after reading that.
I don't know if it was the book, a placebo effect, or something else, but I got better after reading that.
Have you tried strength training? Do you use a tenting split keyboard and vertical mouse? Do you have a good chair, decent posture, and a desk at the proper height? Those things solved my RSI.
>Having a BCI would improve my quality of life immensely.
Current BCI tech is nowhere near the fidelity of non-BCI control methods (e.g. eye tracking). If you're serious about it, look into them.
Current BCI tech is nowhere near the fidelity of non-BCI control methods (e.g. eye tracking). If you're serious about it, look into them.
> RSI
https://workrave.org/
Also, check your desk and chair ergonomics. I find having my elbows on the desk reduces wrist strain.
https://workrave.org/
Also, check your desk and chair ergonomics. I find having my elbows on the desk reduces wrist strain.
You could take a look at the crown by Neurosity: https://neurosity.co/. I was on the waitlist for ctrl-labs as well.
I haven't heard of this thing yet. Do you have one?
i would guess that anything that works for coding/computing use cases is a long ways off. like speech recognition, these things rely heavily on language models...
i suppose we have language models for code now, but i suspect it would still be frustrating- at least in the short term.
but, we shall see...
i suppose we have language models for code now, but i suspect it would still be frustrating- at least in the short term.
but, we shall see...
Wrt to the wristband, the concept and hardware is simple enough that were it not for Facebook nightmare lawyers and patents it could be a cheap maker community staple. Under Facebook, it will fail - nobody wants Facebook tracking literally everything they type.
Huh? This technology (using electromyography to detect hand/arm movements) has been around for at least 6 years (https://www.pcmag.com/reviews/myo-gesture-control-armband) but has not become a "community staple". The company producing the armband actually shelved the product because it was so bad and focused their efforts on AR glasses instead.
I look forward to trustless biometrics storage/processing
This is a really hard problem. NIRS is already difficult; very weak signal, limited to <2 cm of cortical surface, and haemodynamic response is 5-7 seconds.
I think it's definitely a useful imaging technique for certain (highly specific) tasks. But to detect language? They would need many, many more source/detector pairs, considering the vast swathes of the brain partially responsible. Language is already very fuzzy within fMRI.
I think it's definitely a useful imaging technique for certain (highly specific) tasks. But to detect language? They would need many, many more source/detector pairs, considering the vast swathes of the brain partially responsible. Language is already very fuzzy within fMRI.
It reminds me of a former colleague joking about NIRS being "one of the most complex ways to generate white noise" during an experiment :)
I guess there is also the possibility that they just changed their mind and didn't want it to be usable consumer technology at this time? Maybe i am being overly suspicious, but i guess it seems to be at least a little strange that DARPA would put someone on the task who says "it's closer than you realize" and then do a complete 180.
Also, i don't really follow the tech, but i thought i remembered seeing (on HN maybe?) positive reports of actually useful BCI prototypes. Maybe those were overblown or limited, though.
Also, i don't really follow the tech, but i thought i remembered seeing (on HN maybe?) positive reports of actually useful BCI prototypes. Maybe those were overblown or limited, though.
I studied a PhD in this field (dropped out after one year).
There are useful prototypes out there ("useful" as in someone with motor neurone disease moving a computer cursor on a text to speech machine), but all of them use invasive, penetrative electrodes that fail after a matter of months as the body rejects them.
The less invasive prototypes have seen no real breakthroughs in the last decade and a half. Only a small percentage of people can use them with any real accuracy, nobody knows why and frankly nobody's doing the research on it.
All the grant money goes to people throwing random ML models at standard datasets and patting themselves on the back when it gets a 2% more accurate fit on one (and only one) set.
Couple this with the fact that technology relying on other biosignals is yielding amazing progress (mainly the translation of voluntary muscle/tendon movement into control signals for a computer, like what Stephen Hawking had but more recently prosthetic hands with dinner and finger control), I suspect that most of the research into BCIs will be put on ice until we have a more complete understanding of how the brain works or can no longer make progress with other control technologies.
There are useful prototypes out there ("useful" as in someone with motor neurone disease moving a computer cursor on a text to speech machine), but all of them use invasive, penetrative electrodes that fail after a matter of months as the body rejects them.
The less invasive prototypes have seen no real breakthroughs in the last decade and a half. Only a small percentage of people can use them with any real accuracy, nobody knows why and frankly nobody's doing the research on it.
All the grant money goes to people throwing random ML models at standard datasets and patting themselves on the back when it gets a 2% more accurate fit on one (and only one) set.
Couple this with the fact that technology relying on other biosignals is yielding amazing progress (mainly the translation of voluntary muscle/tendon movement into control signals for a computer, like what Stephen Hawking had but more recently prosthetic hands with dinner and finger control), I suspect that most of the research into BCIs will be put on ice until we have a more complete understanding of how the brain works or can no longer make progress with other control technologies.
This work was based on a computational breakthrough of light propagation previously thought to be unsolvable.
My take is that Facebook is being a lot more realistic about this than Neurallink is.
Nothing wrong with being lofty, but with the money and manpower that FB has put behind BCI, versus the talent flight from Neurallink – FB will likely be the winner.
A shame, because I don't think anyone else can even come close to really stacking up.
Nothing wrong with being lofty, but with the money and manpower that FB has put behind BCI, versus the talent flight from Neurallink – FB will likely be the winner.
A shame, because I don't think anyone else can even come close to really stacking up.
From the article it seems FB and NL took very different approaches -- Facebook is against cutting skulls.
I don't know where this will land, but we could argue that Facebook is being unrealistic and true consumer BCI does require an implant.
I don't know where this will land, but we could argue that Facebook is being unrealistic and true consumer BCI does require an implant.
> true consumer BCI does require an implant.
Good luck convincing millions of people getting a brain implant.
Good luck convincing millions of people getting a brain implant.
It depends what you mean. They chose very different approaches. Neuralink is much more technically grounded than attempting high bandwidth optical non-invasive decode, and obviously requires implantation.
> talent flight from Neurallink
Who left?
Who left?
Max Hodak, co-founder (the other founder being Musk himself) and president, left the company a couple of months ago.
That's not talent flight. He was likely pushed out. But Elon's companies are a grinder though, one of the cofounders Vannessa Toulosa is now at FB.
Ah I see, there's more than one other cofounder. Silly me, I used the Google info box..
The report in the NEJM on the research:
https://www.nejm.org/doi/full/10.1056/NEJMoa2027540?query=fe...
Neuroprosthesis for Decoding Speech in a Paralyzed Person with Anarthria
METHODS
We implanted a subdural, high-density, multielectrode array over the area of the sensorimotor cortex that controls speech in a person with anarthria (the loss of the ability to articulate speech) and spastic quadriparesis caused by a brain-stem stroke.
Over the course of 48 sessions, we recorded 22 hours of cortical activity while the participant attempted to say individual words from a vocabulary set of 50 words.
We used deep-learning algorithms to create computational models for the detection and classification of words from patterns in the recorded cortical activity. We applied these computational models, as well as a natural-language model that yielded next-word probabilities given the preceding words in a sequence, to decode full sentences as the participant attempted to say them.
RESULTS We decoded sentences from the participant’s cortical activity in real time at a median rate of 15.2 words per minute, with a median word error rate of 25.6%.
In post hoc analyses, we detected 98% of the attempts by the participant to produce individual words, and we classified words with 47.1% accuracy using cortical signals that were stable throughout the 81-week study period.
CONCLUSIONS In a person with anarthria and spastic quadriparesis caused by a brain-stem stroke, words and sentences were decoded directly from cortical activity during attempted speech with the use of deep-learning models and a natural-language model.
(Funded by Facebook and others; ClinicalTrials.gov number, NCT03698149. opens in new tab.)
https://www.nejm.org/doi/full/10.1056/NEJMoa2027540?query=fe...
Neuroprosthesis for Decoding Speech in a Paralyzed Person with Anarthria
METHODS
We implanted a subdural, high-density, multielectrode array over the area of the sensorimotor cortex that controls speech in a person with anarthria (the loss of the ability to articulate speech) and spastic quadriparesis caused by a brain-stem stroke.
Over the course of 48 sessions, we recorded 22 hours of cortical activity while the participant attempted to say individual words from a vocabulary set of 50 words.
We used deep-learning algorithms to create computational models for the detection and classification of words from patterns in the recorded cortical activity. We applied these computational models, as well as a natural-language model that yielded next-word probabilities given the preceding words in a sequence, to decode full sentences as the participant attempted to say them.
RESULTS We decoded sentences from the participant’s cortical activity in real time at a median rate of 15.2 words per minute, with a median word error rate of 25.6%.
In post hoc analyses, we detected 98% of the attempts by the participant to produce individual words, and we classified words with 47.1% accuracy using cortical signals that were stable throughout the 81-week study period.
CONCLUSIONS In a person with anarthria and spastic quadriparesis caused by a brain-stem stroke, words and sentences were decoded directly from cortical activity during attempted speech with the use of deep-learning models and a natural-language model.
(Funded by Facebook and others; ClinicalTrials.gov number, NCT03698149. opens in new tab.)
The project itself (not Facebook cutting funding) was just covered on NPR - https://www.npr.org/sections/health-shots/2021/07/14/1016028....
“ The company says it now plans to open-source the software it developed for brain decoding and also provide access to prototype devices, so other researchers can benefit from its work”
Anyone found this repo ? I would love to see some of their pipeline’s parameters. They ‘should’ have a fine-tuned processing pipeline that’s interesting for different kind of bio-signal sensors.
Anyone found this repo ? I would love to see some of their pipeline’s parameters. They ‘should’ have a fine-tuned processing pipeline that’s interesting for different kind of bio-signal sensors.
[deleted]
[deleted]
There is something similar to this: `Spotify granted patent to use mic to infer emotional state, age, gender, and accent`[0]. Not surprising, given how data hungry most tech companies are now. Without data, they are dead. But how much is really needed to serve ADs? Do we really need to be doing creepy things like 'sentiment analysis' or peering inside the brain? How much access is too much access? I think we are slowly learning the threshold piece by piece.
[0] https://www.privateinternetaccess.com/blog/spotify-granted-p...
[0] https://www.privateinternetaccess.com/blog/spotify-granted-p...
We arnt slowly learning the threshold. It's known, and we passed it a long time ago. People turn off tracking when given the choice and told what is being tracked already. Much less some future peering inside the brain.
[deleted]
BCI is out there until it's not.
Interestingly, I interviewed at one of the many startups who wants to implant thousands of sensors directly into the brain. The only issue is they had the same problem that MIT Lincoln Labs has: they're snobs about hiring only PhD EEs, creating an ideological monoculture.
Interestingly, I interviewed at one of the many startups who wants to implant thousands of sensors directly into the brain. The only issue is they had the same problem that MIT Lincoln Labs has: they're snobs about hiring only PhD EEs, creating an ideological monoculture.
Maybe when putting things into peoples brains, competence is a highly necessary requirement. It's not enough to have "liked coding since high school" and "read a bunch of blogs" and "taught myself javascript" which may suffice for the average web shop, but when opening skulls, a broad and deep education definitely helps. Those crying about gatekeeping tend to be firmly on one side of the gate without putting in the necessary sweat.
I think a team with a diverse set of expertise, skills, and training, will be safer and outperform a team with an ideological monoculture.
Everyone having the same background opens you up to blind spots, that are arguably worse IMO when you're trying to put things in human brains.
Also, PhD EEs != competence.
Everyone having the same background opens you up to blind spots, that are arguably worse IMO when you're trying to put things in human brains.
Also, PhD EEs != competence.
a PHD is a monkey trained to reason about a unknown.
totally agree that heterogeneous teams are vastly superior in avoiding blind spots.
totally agree that heterogeneous teams are vastly superior in avoiding blind spots.
The comment was not about heterogeneous or not. It was about Ph.D. or not. Everything else equal, I'd prefer a heterogeneous bunch of Ph.D.s to have designed and built the thing that's put inside my brain over a similarly heterogeneous bunch of "I've taught this myself in 1 year using youtube". Snobs or not.
Ph.D.s are trained in two things. "Reasoning about an unknown" is one of them. But the particular niche knowledge they acquired while training this is the other. That niche can save my life if the niche matches the thing to be implanted into my brain. No matter how many youtube-educated wannabe-experts take issue with formal education.
(I have many issues with the academic system, but this kind of critisism is ridiculous.)
Ph.D.s are trained in two things. "Reasoning about an unknown" is one of them. But the particular niche knowledge they acquired while training this is the other. That niche can save my life if the niche matches the thing to be implanted into my brain. No matter how many youtube-educated wannabe-experts take issue with formal education.
(I have many issues with the academic system, but this kind of critisism is ridiculous.)
Would you rather have a team of 10 PhDs and 3 "self taught hackers" or 13 PhDs?
It might just be me, but I actually feel more comfortable with the former.
If you ONLY hire people from specific backgrounds, you get blind spots. That's dangerous. Excluding anyone without a PhD should not be expected to improve safety.
It might just be me, but I actually feel more comfortable with the former.
If you ONLY hire people from specific backgrounds, you get blind spots. That's dangerous. Excluding anyone without a PhD should not be expected to improve safety.
That makes it sound like Ph.D. graduates are a homogenuous bunch. That's nonsense. Somebody with a Ph.D. degree is just that -- a person with a certain degree certifying a certain competence. Apart from that, it does not say anything. Not what background the person has, how they think, what they like. It's part of a Ph.D. degree to aquire knowledge and skills alone. "Self-taught" if you will.
The distinction you make does not exist in that sense. The only actual difference is that the "extra" 3 Ph.D. graduates have had a certification for a somewhat structured education in some research field, while the youtoube fans don't. So, everything else equal, that group has an advantage.
The distinction you make does not exist in that sense. The only actual difference is that the "extra" 3 Ph.D. graduates have had a certification for a somewhat structured education in some research field, while the youtoube fans don't. So, everything else equal, that group has an advantage.
Interesting - out of curiosity a few years ago, I glanced at neuralink's hiring site and was surprised to see the same. I'm sure these folks are very qualified at what they do, but if we're hoping to put these devices in humans, at some point all of the models and "hard" data will have to be matched up to something messier - disease characteristics, quality of life, etc. How many PhD EE's does it take to come up with an operational definition of things like "Depression" or "Multiple Sclerosis symptoms" so you can just "turn this activity down"? Good luck...
Unless you have years of experience at IC design companies, where would you even get the experience for IC design and implantation without a PhD?
Oh noes… anyway.
Plot twist: FB stopped the funding because it is perfect now.
Makes sense considering most of facebook's users lack properly functioning brains to begin with.
I dropped out of a PhD back in 2018 all about the application of Machine Learning to Neurosignal Decoding, and that quote just perfectly sums up everything wrong with the state-of-the-art approach.
A bizarre prompt, an infeasibly boring task, and they somehow magically expect a machine learning to smooth everything over despite the fact that the subject isn't even generating a consistent signal to start with.