Show HN: Gesture controlled Smart Mirror(github.com)
github.com
Show HN: Gesture controlled Smart Mirror
https://github.com/Shinao/SmartMirror
45 comments
Just curious but what do people see the major use of smart mirrors being? I'm personally only in front of a mirror a few minutes a day but I know others are a lot more. Also depending on the mirror re-installation would be quite a pita and so would likely be as bad as in dash systems for aging out. Especially with early rapid iteration.
This looks awesome. Latency in touch/gesture systems is brutal though.
Hi there. I would like to turn your attention to this excellent paper on hand gesture recognition [1]. I ran across it in my foray into hand tracking and gesture recognition with the Kinect. I implemented the Fourier descriptors described in that paper (there's even source code in the paper!) and agree that they are quite effective.
Since you already have a decent looking hand segmentation method, you could simply trace the outline of the segmentation (opencv will give you contours if you ask politely) and generate descriptor vectors with that. Fourier descriptors are rotationally invariant, and you can get scale invariance easily by scaling your hand images to a constant size after segmentation. You can use the descriptors with a variety of ML algorithms, but K nearest neighbors is probably the easiest (and it was I implemented). SVM is probably also a good method. Using ML has the absolutely gigantic upside that you don't have to write code to recognize individual gestures, and it can learn many gestures (I did 5+ no problem).
I am actually working on open sourcing my code (didn't release it immediately because messy code + busy life during the semester), and porting it to Python so that I can use Numpy and Scipy. The original version I wrote in C++, which includes hand tracking (via OpenNI) + static gesture recognition on the Kinect 360 sensor, as well as a CLI for interfacing with the code and building a gesture library. To start tracking, you have to wave vigorously at the camera. In the Python version for the Kinect One sensor, I'm cleaning things up and also implementing my own hand detection and tracking algorithm (based on an unscented Kalman filter) so that I can kill the dependency on OpenNI, which will help calm the vigor of the wave gesture required to initialize OpenNI tracking. I expect good results when the Python version is finished, due to the high quality depth based hand segmentations I am seeing with the new Kinect. I might be able to hook you up with some source code if you don't mind seeing the rough draft and having a tricky install process.
While the older Kinect works with the Fourier method, it is not as good as the Kinect One sensor because it uses a depth reconstruction algorithm that results in jagged edges and merged fingers. The new sensor uses a time of flight sensor which gives very nice accuracy and normal looking edges. Granted, you could get around this problem via skin segmentation on the RGB or IR images that both Kinects also provide.
A camera only system like yours can work well with enough effort, but with a depth image you will almost certainly get better segmentation and detection results with simpler code. Then you can focus on higher level problems like static and dynamic gesture recognition, and design a better UX around those things. This does come at the price of needing a more expensive and larger sensor, though there's a few depth camera options available these days.
Hope this helps! Happy to discuss more if you'd like.
[1] http://www.bu.edu/vip/files/pubs/reports/CCSSM13-04buece.pdf
Edit: re your comment on Python, I have gone the C++ route (as well as just about every other mainstream language) and can honestly say that Python is a much easier development environment than what you will find in C++. Numpy and Scipy are where it is at. Even after completing two major software projects in C++, I have almost an order of magnitude greater productivity in Python, largely because these libraries and the incredibly readable syntax. Also they are blazing fast - you will be hard pressed to write faster C++ than the C code that powers Numpy. In my experience the algorithms you use and your system architecture will be a much larger factor than your language of choice. I choose what gives me the highest efficiency in development because I can focus more on nailing the architecture, algorithms and building necessary tooling. If you have never worked with C++, I assure you, the darkness that is C++ compilation and linking hell is not for the faint hearted.
Since you already have a decent looking hand segmentation method, you could simply trace the outline of the segmentation (opencv will give you contours if you ask politely) and generate descriptor vectors with that. Fourier descriptors are rotationally invariant, and you can get scale invariance easily by scaling your hand images to a constant size after segmentation. You can use the descriptors with a variety of ML algorithms, but K nearest neighbors is probably the easiest (and it was I implemented). SVM is probably also a good method. Using ML has the absolutely gigantic upside that you don't have to write code to recognize individual gestures, and it can learn many gestures (I did 5+ no problem).
I am actually working on open sourcing my code (didn't release it immediately because messy code + busy life during the semester), and porting it to Python so that I can use Numpy and Scipy. The original version I wrote in C++, which includes hand tracking (via OpenNI) + static gesture recognition on the Kinect 360 sensor, as well as a CLI for interfacing with the code and building a gesture library. To start tracking, you have to wave vigorously at the camera. In the Python version for the Kinect One sensor, I'm cleaning things up and also implementing my own hand detection and tracking algorithm (based on an unscented Kalman filter) so that I can kill the dependency on OpenNI, which will help calm the vigor of the wave gesture required to initialize OpenNI tracking. I expect good results when the Python version is finished, due to the high quality depth based hand segmentations I am seeing with the new Kinect. I might be able to hook you up with some source code if you don't mind seeing the rough draft and having a tricky install process.
While the older Kinect works with the Fourier method, it is not as good as the Kinect One sensor because it uses a depth reconstruction algorithm that results in jagged edges and merged fingers. The new sensor uses a time of flight sensor which gives very nice accuracy and normal looking edges. Granted, you could get around this problem via skin segmentation on the RGB or IR images that both Kinects also provide.
A camera only system like yours can work well with enough effort, but with a depth image you will almost certainly get better segmentation and detection results with simpler code. Then you can focus on higher level problems like static and dynamic gesture recognition, and design a better UX around those things. This does come at the price of needing a more expensive and larger sensor, though there's a few depth camera options available these days.
Hope this helps! Happy to discuss more if you'd like.
[1] http://www.bu.edu/vip/files/pubs/reports/CCSSM13-04buece.pdf
Edit: re your comment on Python, I have gone the C++ route (as well as just about every other mainstream language) and can honestly say that Python is a much easier development environment than what you will find in C++. Numpy and Scipy are where it is at. Even after completing two major software projects in C++, I have almost an order of magnitude greater productivity in Python, largely because these libraries and the incredibly readable syntax. Also they are blazing fast - you will be hard pressed to write faster C++ than the C code that powers Numpy. In my experience the algorithms you use and your system architecture will be a much larger factor than your language of choice. I choose what gives me the highest efficiency in development because I can focus more on nailing the architecture, algorithms and building necessary tooling. If you have never worked with C++, I assure you, the darkness that is C++ compilation and linking hell is not for the faint hearted.
Amazing ressource thank you ! I should have search deeper before my project, clearly not finished reading yet but there are clearly superior methods than I currently have.
My grudge on python was probably linked with the incompatible libs with the version 3. I should have used 2.7 and miss some features with OpenCV in restrospective, but I wanted to learn python at the same time and it seemed a waste of time to begin with an older version. I've had a lot more experience in C++ and I don't mind a few dependencies problem, but it's probably that I'm used to it now.
I would love to see if you have a demo of what you have done (we never think to record our projects but heh you never know), if not a link to the code can be interesting when you aren't busy :)
My grudge on python was probably linked with the incompatible libs with the version 3. I should have used 2.7 and miss some features with OpenCV in restrospective, but I wanted to learn python at the same time and it seemed a waste of time to begin with an older version. I've had a lot more experience in C++ and I don't mind a few dependencies problem, but it's probably that I'm used to it now.
I would love to see if you have a demo of what you have done (we never think to record our projects but heh you never know), if not a link to the code can be interesting when you aren't busy :)
I posted on reddit yesterday but didn't have much feedback, maybe I will have a better chance here : how would you guys have done it ? Do you have any idea for any new widget that I could integrate ?
It would be nice if you could make the screen run Android, and put a shim layer in between, so the proper touch events would be delivered to Android. If you did that you'd have access to the full Android ecosystem, and you wouldn't have to make your own widgets.
Awesome build! If you are looking for some ideas for new widgets my blog would definitely spark some ideas. Mine ran windows 10, so I used a program called 'rainmeter' that just adds widgets to your desktop. I found music visualizers, candles, gifs, and google chrome experiments really cool.
[1] innate.cc
[1] innate.cc
I think this is a really great POC. Do you have plans to create a real product around it?
I see a lot of potential in the whole "put a screen on the wall and interact with it" line of products, either through e-ink or - as in your case - with the one-way mirror/screen combo.
I see a lot of potential in the whole "put a screen on the wall and interact with it" line of products, either through e-ink or - as in your case - with the one-way mirror/screen combo.
Totally agree. IoT has been a big missfire, but you could make a good argument for these kinds of products that don't change too much about the existing use (the mirror is still a mirror) but just add tons of cool IoT features.
I say you kickstarter it as an open-source backed product: produce and sell them in volume and publish the guides to make your own if you want.
I say you kickstarter it as an open-source backed product: produce and sell them in volume and publish the guides to make your own if you want.
Nope, I just wanted to see if I could make it. Biggest problem for this kind of stuff is the power source. I agree I can totally see an e-ink displayin my bathroom/hallway in the future.
Curious if there is information about assembling the hardware.
It would be great to get a tutorial on how to assemble the project and how to use the software. I am also curious to know where the camera was placed on the mirror. The NoIR cameras are quite "raw".
Didn't include any assembly tutorial since there is a lot of DIY Smart Mirror out there. You are right I'm gonna include an how to use the software right now. The camera is black taped behind the mirror so that we can't see it. Integrating into the frame should work too.
I acknowledge the technological feat, but could you tell us about your motivation? What is the place of a smart mirror ?
Obviously having a new cool gadget was my big motivation, but for the useful things : knowing the exterior temperature, the time obviously (I removed my clock), I can play a game if I'm waiting for something for a few minutes, etc. But basically you are right, if I have my phone, it just takes almost as much time for me to do this things.
I think there is something to do around the concept of a smart mirror, because we have been introduced to many prototypes in last 5 years. It's like people want to have it, people want to build it, but no one is sure what it is about. Maybe it should be conceptualized more like a decoration/fashion item...
That's what I thought. You can look at yourself while you browse cinema times?
I'd replace the TV with a `smart mirror`.
The `mirror` would make the living room seem larger and the `smart` would improve the quality of the information provided.
The `mirror` would make the living room seem larger and the `smart` would improve the quality of the information provided.
Just wondering why you rolled your own gesture recognition system rather than using something off-the-shelf like the Leap Motion.
Also, your comments about the Python environment are maybe a little unfair... You have an unusual (and some might argue overly-complex) use case where you're combining it not just with the third-party OpenCV package and wrapper, but also Tornado and Node.js. It would have probably been easier going with either the Python approach or the JavaScript approach rather than combining them.
All in all, pretty neat though.
Also, your comments about the Python environment are maybe a little unfair... You have an unusual (and some might argue overly-complex) use case where you're combining it not just with the third-party OpenCV package and wrapper, but also Tornado and Node.js. It would have probably been easier going with either the Python approach or the JavaScript approach rather than combining them.
All in all, pretty neat though.
I considered Leap Motion but didn't know if it would have worked behind my mirror and in vertical way (Maybe if you have one you call tell me). Also I always like to learn new stuff, OpenCV seemed like something I could use again in the future.
I tried running everything on Node but the OpenCV wrapper is not there yet. Completely right on Python though but while learning it I wasn't interested to go on with it for the front end.
Thanks !
I tried running everything on Node but the OpenCV wrapper is not there yet. Completely right on Python though but while learning it I wasn't interested to go on with it for the front end.
Thanks !
We've tried that here - one-way mirrors, straight glass, plexiglass, projection film - leap motion will NOT work through the materials.
It CAN work vertically with caveats.
It CAN work vertically with caveats.
I used a gesture controller (the name is escaping me) on my smart mirror, placing it on the frame facing outwards/up. The mirror reflected the infrared as well, so it didn't work. Pointing the controller directly away from the mirror worked, but made gesture very awkward.
How did you implement the hand detection? Since your using HSV color space, are your just doing color segmentation for a guessed range of Hue, Saturation, and Value for the color of skin? I have a few ideas on how to really improve the detection since I've worked on skin segmentation before
Oh nice, yes that's exactly what I'm doing, I take the middle of my hand, take the HSR point and increase it's range to get the whole hand. I'm listening :)
Nice! You can implement a real time skin segmentation algorithm using HSV segmented skin as training, and using random forests to do classification, it classifies in real-time with opencv!
Here is a video of my result from only training with one frame of a segmented HSV hand to some random videos: https://vimeo.com/114489035
Also, you can use a kinect or an occipital structure sensor to do segmentation from a certain depth, more hardware but less vision computation.
Here is a video of my result from only training with one frame of a segmented HSV hand to some random videos: https://vimeo.com/114489035
Also, you can use a kinect or an occipital structure sensor to do segmentation from a certain depth, more hardware but less vision computation.
And using a depth sensor may be more robust for hand tracking and gesture recognition.
Thanks for the info, learnt something. What kind of project did you do for skin segmentation ?
I was working on skin segmentation for a research project to it segment in real time for Google Glass! I never did the training with a huge database of ground truths. I want to eventually complete the training right to build a robust skin segmentation alg. I can link the github page(Opencv implementation only), but my code is really messy as I built as a POC in two days.
The google glass app was to use skin segmentation for gesture recognition.
The google glass app was to use skin segmentation for gesture recognition.
Oh nice ! Maybe can be used for translating sign language in real time, that would be amazing, kinda hard to get the depth infos though.
Thanks for the answers and if you ever finish the alg you know where to post it :)
Thanks for the answers and if you ever finish the alg you know where to post it :)
I'll let you know, We should collaborate and make the mirror be interactive like Jarvis! :D. Message me at [email protected]
I don't understand anything you two just said, and it was awesome---this is why i come to HN. Nice work!
What do you recommend for depth sensing ?
We had a student once who used a face detector to first find the face area, and then extracted the skin colour from that region for the subsequent hand tracking.
[deleted]
Thanks for sharing your cool hack. I remember two similar projects a while back that caught HN's attention :
https://news.ycombinator.com/item?id=10204018
https://news.ycombinator.com/item?id=10801430
P.S. I'd be tempted to call it Jarvis. It's fun to swipe away icons in the morning :)
https://news.ycombinator.com/item?id=10204018
https://news.ycombinator.com/item?id=10801430
P.S. I'd be tempted to call it Jarvis. It's fun to swipe away icons in the morning :)
I got the inspiration from these kind of projects, I saw a Jarvis one like a few weeks ago (voice activated), I wanted to make one with a Minority Report style in mind though :)
The most popular voice control/recognition platform for rpi is called jarvis.
I am shocked at how many different DIY versions of a smartmirror has been posted here in HN. I was hoping someone would kickstart something like this, but just saw https://www.kickstarter.com/projects/338193274/mirro-the-wor... didn't really get off the ground. Maybe you can kickstart yours and I can be the first to buy :)
It's probably waiting for the mass production of either e-ink display or screen as flat as a tablet to be more attractive and/or low power demanding.
I'm not attracted by the business side of things, but you go for it ! :)
I considered doing a kickstarter for mine. There have been 3 that I could find on kickstarter. Most fail for the same reasons... they try to do too much, don't do it well, and don't have a compelling use case.
The link seems to be a parody Kickstarter project.