I put sheet music into smart glasses [video](youtube.com)
youtube.com
I put sheet music into smart glasses [video]
https://www.youtube.com/watch?v=j36u2i7PKKE
https://github.com/kevinlinxc/AugmentedChords
71 comments
I am so unbelievably excited for consumer-grade, useful AR.
There was a lot of hype around VR, but for the last 10 years I've been following progress on AR glasses.
The thing about AR is that it has the ability to enhance everything in your daily life, versus VR which is meant to be a separate experience.
Both Meta and Samsung are due to put out consumer AR glasses later this year and I think this might be the first wave of useful, daily-wear glasses we'll see.
Is there anyone who works in the AR space that could comment more?
There was a lot of hype around VR, but for the last 10 years I've been following progress on AR glasses.
The thing about AR is that it has the ability to enhance everything in your daily life, versus VR which is meant to be a separate experience.
Both Meta and Samsung are due to put out consumer AR glasses later this year and I think this might be the first wave of useful, daily-wear glasses we'll see.
Is there anyone who works in the AR space that could comment more?
100% agree.
I've been building smart glasses for over 7 years. First 6 years were in academia because the tech wasn't ready yet, because they were too heavy and battery didn't last long enough.
But in the last year, all-day battery smart glasses have become lightweight enough to be worn all day (see Even Realities G1, Vuzix Z100, etc.).
I believe smart glasses are having their iPhone moment in 2025 + 2026.
We make the smart glasses OS that Kevin used in the video to make this smart glasses app: AugmentOS.org
I've been building smart glasses for over 7 years. First 6 years were in academia because the tech wasn't ready yet, because they were too heavy and battery didn't last long enough.
But in the last year, all-day battery smart glasses have become lightweight enough to be worn all day (see Even Realities G1, Vuzix Z100, etc.).
I believe smart glasses are having their iPhone moment in 2025 + 2026.
We make the smart glasses OS that Kevin used in the video to make this smart glasses app: AugmentOS.org
I want one thing - tell me people's names. I have the unholy ability to say your name several times and have an hour long conversation, and still not remember it next time I see you. People take this so personally that I've started avoiding some social gatherings where I only lightly know people! I'd love to "know" the name of everyone I've met because I'd be so much more comfortable seeing and talking to them again!
Are you me? I have the same issue, when I see someone not in my immediate circle and out of context, say on the street, it takes me some time to recall their name from my jumbled memory.
If there’s context (eg I go to another department at work, or see my child’s friend with their parent) I can get their name easier. But that barrier for being out of context can be difficult to surmount.
I’m curious if you also find yourself having trouble to remembering other names when in conversation (eg what was that politician called, what’s that city name, it begins with an F…)
Usually it’s proper nouns that I have trouble recalling. It’s almost like I need an Anki to refresh my mental DRAM and keep things recallable.
If there’s context (eg I go to another department at work, or see my child’s friend with their parent) I can get their name easier. But that barrier for being out of context can be difficult to surmount.
I’m curious if you also find yourself having trouble to remembering other names when in conversation (eg what was that politician called, what’s that city name, it begins with an F…)
Usually it’s proper nouns that I have trouble recalling. It’s almost like I need an Anki to refresh my mental DRAM and keep things recallable.
Yes I very much have those problems. It's strange because I'm objectively a good engineer, but semantic retrieval is very difficult for me. It has been my entire life.
When I was a kid I did a round of testing for ADHD, and one of the tests is category fluency. Basically they gave me a category (e.g. Breakfast foods) and told me to say as many words in that category as I could in 60 seconds. I failed miserably; the administrator told me it was essentially as bad as people with severe cognitive deficiency. I just haven't worked about it since then! (Except with names!!)
When I was a kid I did a round of testing for ADHD, and one of the tests is category fluency. Basically they gave me a category (e.g. Breakfast foods) and told me to say as many words in that category as I could in 60 seconds. I failed miserably; the administrator told me it was essentially as bad as people with severe cognitive deficiency. I just haven't worked about it since then! (Except with names!!)
God yea please - I’ve wanted this for decades.
Facial recognition, with name, where I know you from, and last time I saw you.
Basically I want the same notes my dental hygenist or optometrist uses to make light conversation with me during a checkup.
Facial recognition, with name, where I know you from, and last time I saw you.
Basically I want the same notes my dental hygenist or optometrist uses to make light conversation with me during a checkup.
The fact that people actually want this level of surveillance in exchange for a tiny convenience is astounding to me.
You must be young. Social well being is the number 1 indicator of happiness, and is indeed correlates with long life. If this one change was enough for someone to be more social and spend more time with a wider group of people, it could have many positive effects. I see this in my life.
I'd be neat to have information displayed while driving a car. A subset of information currently displayed on a dashboard would be an obvious first choice (speed for one).
You could also maybe perhaps tie in a car's knowledge of adjacent vehicles, which is something I've wanted for ages. Since some newer models have some level of awareness about the speed / distance to / relative location of cars around your car, you could eg overlay the speed/acceleration info onto adjacent cars, so you'd know if you'll need to pass or speed up. Seems at least possible since the glasses have awareness of your head's orientation, something missing from any existing windscreen-style HUD system.
You could also maybe perhaps tie in a car's knowledge of adjacent vehicles, which is something I've wanted for ages. Since some newer models have some level of awareness about the speed / distance to / relative location of cars around your car, you could eg overlay the speed/acceleration info onto adjacent cars, so you'd know if you'll need to pass or speed up. Seems at least possible since the glasses have awareness of your head's orientation, something missing from any existing windscreen-style HUD system.
Unlike you i think the potential is rather in business and work. Useful info when needed.
[deleted]
Just bought a pair of the Mentra Mach1 glasses, let's see how they pan out.
Any workaround for those needing prescription glasses?
You can get prescription inserts from Vuzix, but they're pretty bad. If you need a prescription, and want to run AugmentOS, your best bet is to buy the Even Realities G1 instead.
I'm only aware of a handful good use cases
Navigation and speed in a car for example
Navigation and speed in a car for example
This project is cool so we're hoping to arrange with Kevin to do a Show HN about it, so stay, er, tuned!
[deleted]
wow that's a special honor. is there a way to search up the "specially invited" Show HNs?
There's a very incomplete list at https://news.ycombinator.com/invited - incomplete because it only includes posts that technically got an invitation link by email, but there are plenty of others. You can also look at https://news.ycombinator.com/pool (explained at https://news.ycombinator.com/item?id=26998308) and find Show HNs in there. But that list includes many Show HNs that we didn't specifically help with beforehand.
(So I guess you can have either a list of too many or a list of too few )
p.s. If anyone notices really cool work that would be even better if the creator did a Show HN, please let us know at [email protected].
(So I guess you can have either a list of too many or a list of too few )
p.s. If anyone notices really cool work that would be even better if the creator did a Show HN, please let us know at [email protected].
fwiw that link is blocked for us: "Please log in as an administrator."
Whoa that should not be happening! Let me check...
Edit: sorry - I gave the wrong URL. It should have said https://news.ycombinator.com/invited. I've fixed it in the comment above too.
Edit: sorry - I gave the wrong URL. It should have said https://news.ycombinator.com/invited. I've fixed it in the comment above too.
thank you dang!
One of our users documented projecting sheet music onto his smart glasses's display (with a HUD). He did a great job documenting the limitations of 2025's tech, but it gives a great look into what's going to be feasible next year.
Awesome job Kevin!
Awesome job Kevin!
Nice work. From the Github:
> This allows the pianist to not have to turn pages, and more importantly, allows them to see the music and their hands at the same time, which is an unavoidable problem with traditional sheet music.
I could definitely see this being beneficial for beginners. When I lived in a dormitory during uni I often played familiar pieces from memory pretty late on a digital piano (with headphones) in extremely dim lighting so as not to disturb my roommate.
At some point I just stopped having to look down at the keyboard. I play a lot of stride piano as well and that probably conditioned me to just have a sort of musical proprioception for the instrument. And of course, there's numerous examples of unbelievable blind pianists - Stevie Wonder, Ray Charles, Art Tatum, etc.
> This allows the pianist to not have to turn pages, and more importantly, allows them to see the music and their hands at the same time, which is an unavoidable problem with traditional sheet music.
I could definitely see this being beneficial for beginners. When I lived in a dormitory during uni I often played familiar pieces from memory pretty late on a digital piano (with headphones) in extremely dim lighting so as not to disturb my roommate.
At some point I just stopped having to look down at the keyboard. I play a lot of stride piano as well and that probably conditioned me to just have a sort of musical proprioception for the instrument. And of course, there's numerous examples of unbelievable blind pianists - Stevie Wonder, Ray Charles, Art Tatum, etc.
I wanted to say, after a certain level looking at your fingers when playing an instrument becomes the equivalent of looking at your legs when riding the bicycle.
When I start to think too much about what my fingers are doing I will play worse. For if I want to practise a particular part where I get the fingering wrong, sure, but when you play it for real, looking is counterproductive.
Something like this could be great for beginners tho. But simular to automatic guitar tuners I am not sure if you should get into the habit of this technology being around.
When I start to think too much about what my fingers are doing I will play worse. For if I want to practise a particular part where I get the fingering wrong, sure, but when you play it for real, looking is counterproductive.
Something like this could be great for beginners tho. But simular to automatic guitar tuners I am not sure if you should get into the habit of this technology being around.
Counter point: people get too hung up on staring at the sheet. The sheet is just a tool to help you remember what you intend to play. The goal should always be to not need it anymore, and while using the sheet, it's like using a crutch.
The "looking at your fingers" challenge then becomes that you start to play "by eye" instead of "by ear" (or "by feel") which I find is very hard to overcome. Especially when you are improvising.
Though in a sense "by sheet" is just as bad.
The "looking at your fingers" challenge then becomes that you start to play "by eye" instead of "by ear" (or "by feel") which I find is very hard to overcome. Especially when you are improvising.
Though in a sense "by sheet" is just as bad.
I am not a big sheet player myself, I love to improvise and play by memory, so when I said "don't look at the fingers", I didn't mean "instead look at the sheet".
Look at the audience, out of the window, into nothingness — or even close your eyes. As long as you're there.
Look at the audience, out of the window, into nothingness — or even close your eyes. As long as you're there.
> The goal should always be to not need it anymore, and while using the sheet, it's like using a crutch.
Uh, why? Lots of pros use sheet music, especially for complex pieces. I’ve never heard of an orchestra conductor insisting everyone be off-book.
It’s one thing to memorize pop songs or whatever, but nobody is out there shaming people for not memorizing Rachmaninov
Uh, why? Lots of pros use sheet music, especially for complex pieces. I’ve never heard of an orchestra conductor insisting everyone be off-book.
It’s one thing to memorize pop songs or whatever, but nobody is out there shaming people for not memorizing Rachmaninov
People who are good at playing a technically hard piece essentially know it by heart. They couldn't play it on sight, they have invested significant time to be able to play it. It's just the few remaining percent of reminders and the comfort of having the sheet "just in" case as well as the being-used-to-it factor that makes them stare at the sheet. But if they can play it really well then they don't _actually_ need it. Any pro musician will tell you this.
The orchestra setting has the extra requitement that the sheet is a tool for communication. "The figure in bar 83" is not a term you have gained an intuitive understanding for, but is needed to communicate in an orchestra setting. The soloist though often times plays by heart, at least during performances, so as not to get distracted / get tunnel vision.
The orchestra setting has the extra requitement that the sheet is a tool for communication. "The figure in bar 83" is not a term you have gained an intuitive understanding for, but is needed to communicate in an orchestra setting. The soloist though often times plays by heart, at least during performances, so as not to get distracted / get tunnel vision.
The soloist had their concerto memorized (and they’ve probably performed it dozens of times before). But the members of the orchestra are responsible for playing hundreds of pages of music a week. They’ve practiced any particularly exposed or technical sections, but otherwise they’re basically sight reading for the performance.
How many people still look at the keyboard when typing? At some point, you just don't need to look at whatever it is your doing. Also, at some point, you memorize the music if it's something you are playing enough. What level of pianist is reading sheet music and looking at their hands at the same time.
> I just stopped having to look down at the keyboard
Maybe the next step is an app for people who don't read sheet music; it would light up the keys on the keyboard that you need to press, when you look at it...?
Same for guitar, highlight where to put your fingers on a fret for each chord.
Maybe the next step is an app for people who don't read sheet music; it would light up the keys on the keyboard that you need to press, when you look at it...?
Same for guitar, highlight where to put your fingers on a fret for each chord.
Various products have done this over the years. Forget app, actual keyboards and guitars that light up. It’s not a good way to learn.
It's true that it's not a good way to learn, but it's fun for a little while.
That's why an app on glasses for this is better than a complete alternative instrument; the app is much cheaper and should work with any real instrument.
That's why an app on glasses for this is better than a complete alternative instrument; the app is much cheaper and should work with any real instrument.
An app on glasses would also not be a great way to learn. There’s a reason we don’t color code piano keys or right the names of the notes on them.
> the app is much cheaper and should work with any real instrument.
Maybe if your instrument has keys or frets. Even if it was a good way to learn, it wouldn’t do much good for a trumpet.
> the app is much cheaper and should work with any real instrument.
Maybe if your instrument has keys or frets. Even if it was a good way to learn, it wouldn’t do much good for a trumpet.
> There’s a reason we don’t color code piano keys or write the names of the notes on them.
When's the last time you learned to play piano? Yes we do. We take off those training wheels at some point but when learning to play for the first time, especially as a child, tools to help learning like the aforementioned do get used.
When's the last time you learned to play piano? Yes we do. We take off those training wheels at some point but when learning to play for the first time, especially as a child, tools to help learning like the aforementioned do get used.
A couple of times. I learned initially as a kid, then in college I took two semesters of piano, then in the last year I’ve delved heavily into keyboard playing.
I taught my 5 year old where C was and it took about 2 minutes.
> We take off those training wheels
Training wheels are an apt metaphor because they are a similarly wrong way to learn to bike. You get quicker initial results, but have to unlearn to use those crutches later on.
I taught my 5 year old where C was and it took about 2 minutes.
> We take off those training wheels
Training wheels are an apt metaphor because they are a similarly wrong way to learn to bike. You get quicker initial results, but have to unlearn to use those crutches later on.
What qualifies you to speak to the usefulness of techniques you've never personally used? Being able to learn without them doesn't mean they don't help in learning.
> I taught my 5 year old where C was and it took about 2 minutes.
Just 87 more keys to go! At 2 minutes per, only 2 hours, 52 minutes until they're playing Mozart's. Now if only that's how it actually works.
> I taught my 5 year old where C was and it took about 2 minutes.
Just 87 more keys to go! At 2 minutes per, only 2 hours, 52 minutes until they're playing Mozart's. Now if only that's how it actually works.
It’s not even a good way to play. The delay between seeing the cue, and moving to play it, completely ruins the flow of playing (and listening to) the music - it introduces too many hiccups in timing.
For me, it's still easy to mess up for complex sections if I'm not peeking at the keyboard every so often. Its true that muscle memory takes over after you reach a level of familiarity but not quite to the extent of biking
TIL about their glasses https://mentra.glass/
great video editing, OP. loved the playthru at the end with the text. you have real talent here, keep giong
great video editing, OP. loved the playthru at the end with the text. you have real talent here, keep giong
Thank you, the typewriter effect is surprisingly hard, requires a pretty complicated expression in after effects
i feel like someone could make "new aftereffects" just making the common things that people do easier
got any more of these effects you'd put in that category?
got any more of these effects you'd put in that category?
Another huge time sink is rotoscoping, e.g. in my video I rotoscope the legs of the piano bench and its a cool effect but it took almost an hour for those few seconds. AI has come far here, but afaik there's no great, free workflow yet. Would love to Docker compose something on my computer, open a webapp, drag my clip in and export segmented videos.
Fun, did that 8 years ago with the original Hololens https://www.youtube.com/watch?v=E6cBX4t2kX0
Super cool PoC. Also an advertisement for the value of local processing over cloud.
This is really awesome!
I was surprised about using dilation. I would have expected music21 to support rendering to a certain resolution/dpi setting directly and avoid rescaling the images. But from the music21 documentation this is not obvious how to do it. Rendering music to a low dpi screen nicely (pixel perfect) could circumvent some of the hardware limitations in the mid term.
I was surprised about using dilation. I would have expected music21 to support rendering to a certain resolution/dpi setting directly and avoid rescaling the images. But from the music21 documentation this is not obvious how to do it. Rendering music to a low dpi screen nicely (pixel perfect) could circumvent some of the hardware limitations in the mid term.
This is a really cool project.
From the end of the video:
> I had the bars of music auto-sending at a preset interval. The pedals, instead of flipping bars, temporarily pause the flipping or speed it up, in case I'm desynced from the glasses.
That's how teleprompter apps work. Of course the difference is that when speaking you can pause a little if you get desynced, while with music you're like "on a train" and if you pause, it shows. But having an interval is not shocking.
Maybe the problem with this is that typically, sheet music resolution is not constant -- if there are many short notes it will result in a larger space on the page (a larger bitmap) than if there are few long notes.
So maybe an approach is to send a fixed number of bars, regardless of their actual size, so that the interval can have a constant relation with the tempo of the piece?
> My dream smart glasses would just listen to the performance and automatically flip bars
Couldn't the phone do that? The phone is already the part doing most of the work.
From the end of the video:
> I had the bars of music auto-sending at a preset interval. The pedals, instead of flipping bars, temporarily pause the flipping or speed it up, in case I'm desynced from the glasses.
That's how teleprompter apps work. Of course the difference is that when speaking you can pause a little if you get desynced, while with music you're like "on a train" and if you pause, it shows. But having an interval is not shocking.
Maybe the problem with this is that typically, sheet music resolution is not constant -- if there are many short notes it will result in a larger space on the page (a larger bitmap) than if there are few long notes.
So maybe an approach is to send a fixed number of bars, regardless of their actual size, so that the interval can have a constant relation with the tempo of the piece?
> My dream smart glasses would just listen to the performance and automatically flip bars
Couldn't the phone do that? The phone is already the part doing most of the work.
Congratulations for the project and for winning the hackathon, nicely done!
I am looking for hackable smart glasses with camera which doesn't rely upon any proprietary service to work, Mentra seems to have a camera version but this video seems to suggest that we need to use their service all the time?
I am looking for hackable smart glasses with camera which doesn't rely upon any proprietary service to work, Mentra seems to have a camera version but this video seems to suggest that we need to use their service all the time?
Mentra Live is indeed a camera + speakers + microphone pair of lightweight smart glasses - that you can build for.
Mentra Live runs AugmentOS, so you can control all the I/O (camera, speakers, mic) in your own app with the AugmentOS SDK.
Regarding use the backend service all the time: Most apps/developers connect through official AugmentOS.org servers and focus on building their apps, but you can self-host your own backend if you want.
Mentra Live runs AugmentOS, so you can control all the I/O (camera, speakers, mic) in your own app with the AugmentOS SDK.
Regarding use the backend service all the time: Most apps/developers connect through official AugmentOS.org servers and focus on building their apps, but you can self-host your own backend if you want.
Really nice project.
Would be interesting to dive a bit deeper where this 3s latency comes from. I assume the bitmaps have been pregenerated so I guess it's just the turnaround time when accessing the AugmentOS servers?
Would be interesting to dive a bit deeper where this 3s latency comes from. I assume the bitmaps have been pregenerated so I guess it's just the turnaround time when accessing the AugmentOS servers?
The 3 seconds of latency is writing a bitmap image from the phone to the smart glasses over BLE. It's very fast to send text, but slow to send bitmaps.
For text, everything is fast - the AugmentOS servers introduce <350ms round trip latency in most places/countries.
In next-generation glasses, this will be a lot faster because of better bitmaps handling, BMP encoding schemes, and also pre-loading of BMPs.
For text, everything is fast - the AugmentOS servers introduce <350ms round trip latency in most places/countries.
In next-generation glasses, this will be a lot faster because of better bitmaps handling, BMP encoding schemes, and also pre-loading of BMPs.
A lot of the time is from the glasses themselves, but apparently Mentra talked to the Even CEO and next gen bitmaps will be much better
All these glasses have so many layers of abstraction I don't want between something I develop and the display.
Let me connect via bluetooth direct to the glasses with anything and just tx/rx via a serial port and some low level protocol to get pixels/text on the screen.
This is also the only way I'd be able to buy a pair and feel safe it won't be able to be bricked in 2 years when some company shuts their server down and ends support.
Let me connect via bluetooth direct to the glasses with anything and just tx/rx via a serial port and some low level protocol to get pixels/text on the screen.
This is also the only way I'd be able to buy a pair and feel safe it won't be able to be bricked in 2 years when some company shuts their server down and ends support.
[deleted]
AugmentOS is open source, so feel free to self-host, or even reference our communicator code in our repo to interact with the glasses directly via BLE.
This is an inferior means of development, however.
By going through AugmentOS, you get a much easier development experience, compatibility with multiple pairs of glasses (through both iOS and Android), and the ability to run multiple glasses apps simultaneously.
This is an inferior means of development, however.
By going through AugmentOS, you get a much easier development experience, compatibility with multiple pairs of glasses (through both iOS and Android), and the ability to run multiple glasses apps simultaneously.
Check out brilliant.xyz
That’s a fabulous project video!
Cool project! I have a suggestion - since the processing is done on a moderately powerful laptop anyway, is it possible to bypass the foot pedal and use audio (from laptop or glasses) to predict when to switch to the next bar? I assume it will be a complicated but would trying to match the FFT series to the sheet music pitch data work (or would harmonics cause major headcahes?)
Music transcription has been around for decades, algorithm-wise. In principle this one should actually be easier since it doesn't need to transcribe from scratch but "only" find where in a given sheet you are most likely to currently be.
(Next step: evaluate afterwords and point out mistakes)
(Next step: evaluate afterwords and point out mistakes)
This is great!!!
A lot of digital pianos have midi out, (there was a midi recording tool posted here months ago by another HN member) I wonder if you could use that midi signal to keep what you see in sync with your playing to drive page turns? You could even add a karaoke like highlight to show the note being played.
A lot of digital pianos have midi out, (there was a midi recording tool posted here months ago by another HN member) I wonder if you could use that midi signal to keep what you see in sync with your playing to drive page turns? You could even add a karaoke like highlight to show the note being played.
Really awesome project! I’m reminded of something else in the AR/music space from a few years back. Someone made a VR passthrough app to project synthesia-style keyboard overlays onto your actual piano keys. Always cool to see what new hardware can enable. Congrats!
What a great idea. I am a musician and use an iPad for my scores. It would be wonderful to replace my glasses with a pair that can display the music.
I noticed the iron ring on your pinkie -- Canadian engineer?
I noticed the iron ring on your pinkie -- Canadian engineer?
Correct!
Nice, I recently had a very similar idea. I bought the Vusix Z100 from Amazon japan to do this kind of thing. They also run AugmentOS (supposedly? iOS doesn't seem supported yet) so I'll try yours out.
Okay, I think I'm finally sold that AR has useful applications. As non-gammer, I've sort of mostly written it off. Cool stuff.
Wow can’t wait for v2!
That’s super cool.
Now that’s an awesome hackathon project! Exciting to see that smartglasses are finally getting to an interesting place.
cool~
[deleted]