A Kenyan Engineer Who Created Gloves That Turn Sign Language into Speech(becauseofthemwecan.com)
becauseofthemwecan.com
A Kenyan Engineer Who Created Gloves That Turn Sign Language into Speech
https://www.becauseofthemwecan.com/blogs/culture/meet-the-kenyan-engineer-who-created-gloves-that-turn-sign-language-into-audible-speech
25 comments
You are absolutely correct that while the solutions don’t work that is only part of the problem - the fact that engineers think they can come in and „fix“ communication for the deaf community is a bigger issue. Directly involving your users (as you did) is so important when designing for someone with different needs and preferences. I hope other well meaning talented engineers read your comment and will take it to heart.
By reading your comment and making that very clear, I did!
Taken to <3
I think more people should take it to heart. If you want to influence cultures for the better, you need to interact with them. It's obvious when you know it, less obvious when you don't.
Taken to <3
I think more people should take it to heart. If you want to influence cultures for the better, you need to interact with them. It's obvious when you know it, less obvious when you don't.
> Sign Language gloves are an interesting idea, but they don't work. Sign language relies heavily on facial expressions and body language beyond the hands.
For clarity, could you give an example of a sign that changes to mean a different word or phrase depending on facial expression?
> But deaf people aren't actually that keen on these solutions
People often aren't keen on solutions that are either still nascent or that they haven't given a chance yet. My mom wasn't keen on navigating with a GPS until she had one in her car and it changed her life. Not being keen is not on its own a great reason to dismiss a technology. The reason matters.
IMO a more meaningful statement might be to say that the obvious alternative is simply a small keyboard, which already exists.
For clarity, could you give an example of a sign that changes to mean a different word or phrase depending on facial expression?
> But deaf people aren't actually that keen on these solutions
People often aren't keen on solutions that are either still nascent or that they haven't given a chance yet. My mom wasn't keen on navigating with a GPS until she had one in her car and it changed her life. Not being keen is not on its own a great reason to dismiss a technology. The reason matters.
IMO a more meaningful statement might be to say that the obvious alternative is simply a small keyboard, which already exists.
Questions are almost entirely made with the face (by raising the eyebrows) as one example, which is ASL specific.
I think technology might someday be able to understand ASL and other signed languages (every region/area has their own signed language, generally). But it will be long after computers can easily understand spoken/written languages, which they can only sort of do now and even then very rudimentary currently.
Languages are very complex, and signed languages are no different, but add extra complexity, because you have to also handle facial expressions and hand/arm AND body movements (the whole body moves in ASL to delineate different contexts as another example.)
I for instance just carry around an iPad with a text app (like Sorenson's Buzzcards) which make communicating with hearing people way easier, and save paper.
I think technology might someday be able to understand ASL and other signed languages (every region/area has their own signed language, generally). But it will be long after computers can easily understand spoken/written languages, which they can only sort of do now and even then very rudimentary currently.
Languages are very complex, and signed languages are no different, but add extra complexity, because you have to also handle facial expressions and hand/arm AND body movements (the whole body moves in ASL to delineate different contexts as another example.)
I for instance just carry around an iPad with a text app (like Sorenson's Buzzcards) which make communicating with hearing people way easier, and save paper.
> Questions are almost entirely made with the face (by raising the eyebrows) as one example, which is ASL specific.
Thank you for the clear example. I think this context really helps people understand the conversation better.
Thank you for the clear example. I think this context really helps people understand the conversation better.
I feel like a software and camera approach can work for this then, without dedicated hardware.
The other day I saw a blind person who appeared to be navigating using his phone camera and headphones. Saw him cross the road, etc.
Did you notice whether he appeared to be talking to someone? I wonder if he was using a GPS-based navigation app or a remote assistance service like Aira (https://aira.io/).
Yes in fact he was talking to someone. That's amazing!
what made you predict $600k/btc in 2011
More importantly, there are far more efficient systems like typing on a phone.
Still, I think it's a wonderful idea, wish I'd thought of it. Bravo I say.
Isn't it empirically not a good idea? Deaf people can't use it, it can't work, as sign is more than just the hand movements. It's like using an IDE made by a cello player that uses musical notes instead of letters to write code.
Seems like a fun hardware hack, but if he wants his niece to be able to communicate effectively he should probably just learn frickin sign language. It's unlikely that these can read a vocabulary, just the alphabet, which is incredibly limiting.
So, cool, looks like fun to hack on, almost certainly not newsworthy.
So, cool, looks like fun to hack on, almost certainly not newsworthy.
This is such a negative attitude. A person tried something new in the accessibility space, so what if you think it's trivial.
There's a long history of folks trying "something new" in the accessibility space and discovering that if they'd have talked to the people with impairments they're trying to help that there are significant problems with the approach.
As you can imagine, this external savior thing can be pretty frustrating to the people who actually live their lives with a particular impairment.
As you can imagine, this external savior thing can be pretty frustrating to the people who actually live their lives with a particular impairment.
I really want to play around with sensors and create projects similar to this. I am fascinated with sensors.
Is this something that is possible with arduino or raspberry pi? Or is there something else I should look into?
Is this something that is possible with arduino or raspberry pi? Or is there something else I should look into?
You should be able to do it with a camera, and tensorflow and traing on images with gestures.
Leap Motion skd has something included to get a hand skeleton from the data, could be very useful for something like this.
So you have sensors all over the fingers, use that information to get the current gesture vector, then find the basis vector that makes up the majority of the current gesture, look up the word in a Map(gesture -> word), then pass the word to a TTS engine. That'd be a toy model. What they actually do is probably more nuanced and precise.
Even though this isn't going to be useful for interacting with deaf community, if you can turn gestures into words easily, I feel military might be interested. Also possibly divers, and other people working in conditions where they can't or shouldn't speak.
Can someone do it the other way? UC Berkeley was forced to take down countless hours of recorded lectures because they were violating federal law by not having a provision for blind or deaf people to listen to them.
UC Berkeley was the subject of a Justice Department investigation which stated they weren't in compliance with the law, but never actually sued or subject to a court order to take down anything. They made a unilateral decision.
They decided not to pay for captioning, image enhancement or audio descriptions for their lectures for old content, even though the justice department letter said:
"UC Berkeley is not, however, required to take any action that it can demonstrate would result in a fundamental alteration in the nature of its service, program or activity or in undue financial and administrative burdens".
They could arguably have fulfilled their requirements by working to update material based on actual demand, and having a process to request updating such material. Court ordered consent degrees are negotiated and approved by a judge, and they could have attempted to negotiate something to address the complaint.
UC Berkeley is a public institution, with expenses paid for in large part by federal and state funds, and those funds come with requirements not to discriminate on the basis of disability, and instead of actually addressing that discrimination in a meaningful way for their content, they chose to simply stop providing the service. You can have your opinion on if that's the right decision, but they weren't forced to make one decision or another.
And per UC Berkeley, it wasn't just the Justice department letter, but also online "limited use" and to better protect instructor intellectual property from 'pirates'". https://news.berkeley.edu/2017/03/01/course-capture/
They decided not to pay for captioning, image enhancement or audio descriptions for their lectures for old content, even though the justice department letter said:
"UC Berkeley is not, however, required to take any action that it can demonstrate would result in a fundamental alteration in the nature of its service, program or activity or in undue financial and administrative burdens".
They could arguably have fulfilled their requirements by working to update material based on actual demand, and having a process to request updating such material. Court ordered consent degrees are negotiated and approved by a judge, and they could have attempted to negotiate something to address the complaint.
UC Berkeley is a public institution, with expenses paid for in large part by federal and state funds, and those funds come with requirements not to discriminate on the basis of disability, and instead of actually addressing that discrimination in a meaningful way for their content, they chose to simply stop providing the service. You can have your opinion on if that's the right decision, but they weren't forced to make one decision or another.
And per UC Berkeley, it wasn't just the Justice department letter, but also online "limited use" and to better protect instructor intellectual property from 'pirates'". https://news.berkeley.edu/2017/03/01/course-capture/
I may have written too much. TLDR: This is typing english letters with hand gestures for each letter, not sign language translation which is HARD.
There have been devices like this since the 80s, because it basically does hand shapes, not signs (which have movement, grammar, spatial references, body placement references and other aspects that a hand and wrist device won't capture).
This looks suitable for fingerspelling only (Think, instead of words, spelling out everything you want to say using the letters of the alphabet, and you'll get why this isn't a full solution, though you should understand that sign languages are discrete languages with their own grammar and syntax not just english words as signs, so... I guess imagine spelling out what you want to say but it's transliterated french and using latin declensions or something).
Better then nothing, sure, but hardly translating sign language.
Can you do it with a camera and machine learning? I very much doubt it. I am not sure current gesture segmentation methods could pick out discrete signs or even hand shapes in different orientations, given that there isn't a neutral state one returns between "words".
Then you need to do gesture classification, differentiating based on handshapes (which can transform during a sign, with the nature of the transition, as well as it's speed, the size of the over all gesture, incorporated facial language etc is meaningful, and can be modified by previous or future signs, spatial designations etc), and THEN you need to do stateful real time machine language translation from the decoded signs into English (or Swahili possibly), because a nations sign language isn't english (or whatever your local language is) with the words signed, it's it's own thing, and there may not be a one to one correspondence between words.
In ASL (and I can't speak to other sign languages), you often put the subject of the sentence first (IE instead of saying, "who's the new guy in the purple tie?" the grammar is more like "GUY THERE, WEARING TIE PURPLE, NEW, WHO?") And that's something simple, where gender is included explicitly, no past or future tenses etc. Of course there's also code switching, and formalized systems for conveying English like Signing Exact English.
Oh, did I mention that no sign language has a written form? Linguists and others can create code designations based on what they see as sign word correspondences, but the same sign can be modified, such that the sign you use for a pretty face can be modified to indicate degree of beauty, so the sign for gorgeous and the sign for pretty need to be differentiated, by software.
Oh and if you got it working for one country... you got it working for one country. British sign language and american sign language are radically different, and basically every country has it's own sign language, though they may have some shared roots. Even in the US, because of segregated schools for the deaf, there's a black ASL dialect.
Anyway, we might have the technology at this point to do sign language translation, but probably not, but even if we did, the funding to get it from working on set sign language phrases in an MIT or Carnegie Melon Lab to something that can actually translate sign language isn't there.
If someone wanted to start camera classification for gestures, I'd start with recognizing flipping the bird fast enough to blur it in a live video feed, because people might actually pay for that.
There have been devices like this since the 80s, because it basically does hand shapes, not signs (which have movement, grammar, spatial references, body placement references and other aspects that a hand and wrist device won't capture).
This looks suitable for fingerspelling only (Think, instead of words, spelling out everything you want to say using the letters of the alphabet, and you'll get why this isn't a full solution, though you should understand that sign languages are discrete languages with their own grammar and syntax not just english words as signs, so... I guess imagine spelling out what you want to say but it's transliterated french and using latin declensions or something).
Better then nothing, sure, but hardly translating sign language.
Can you do it with a camera and machine learning? I very much doubt it. I am not sure current gesture segmentation methods could pick out discrete signs or even hand shapes in different orientations, given that there isn't a neutral state one returns between "words".
Then you need to do gesture classification, differentiating based on handshapes (which can transform during a sign, with the nature of the transition, as well as it's speed, the size of the over all gesture, incorporated facial language etc is meaningful, and can be modified by previous or future signs, spatial designations etc), and THEN you need to do stateful real time machine language translation from the decoded signs into English (or Swahili possibly), because a nations sign language isn't english (or whatever your local language is) with the words signed, it's it's own thing, and there may not be a one to one correspondence between words.
In ASL (and I can't speak to other sign languages), you often put the subject of the sentence first (IE instead of saying, "who's the new guy in the purple tie?" the grammar is more like "GUY THERE, WEARING TIE PURPLE, NEW, WHO?") And that's something simple, where gender is included explicitly, no past or future tenses etc. Of course there's also code switching, and formalized systems for conveying English like Signing Exact English.
Oh, did I mention that no sign language has a written form? Linguists and others can create code designations based on what they see as sign word correspondences, but the same sign can be modified, such that the sign you use for a pretty face can be modified to indicate degree of beauty, so the sign for gorgeous and the sign for pretty need to be differentiated, by software.
Oh and if you got it working for one country... you got it working for one country. British sign language and american sign language are radically different, and basically every country has it's own sign language, though they may have some shared roots. Even in the US, because of segregated schools for the deaf, there's a black ASL dialect.
Anyway, we might have the technology at this point to do sign language translation, but probably not, but even if we did, the funding to get it from working on set sign language phrases in an MIT or Carnegie Melon Lab to something that can actually translate sign language isn't there.
If someone wanted to start camera classification for gestures, I'd start with recognizing flipping the bird fast enough to blur it in a live video feed, because people might actually pay for that.
I worked with these folks for a while. I also work with the deaf community.
Sign Language gloves are an interesting idea, but they don't work. Sign language relies heavily on facial expressions and body language beyond the hands.
This was also tried here before: https://www.huffingtonpost.com/entry/navid-azodi-and-thomas-...
But deaf people aren't actually that keen on these solutions, as I found out when I proposed this to some of them myself.