Scribble – Convert handwriting into digital text(getscribblenow.com)
getscribblenow.com
Scribble – Convert handwriting into digital text
http://getscribblenow.com/
63 comments
Thanks for the comment!
The algorithm works on a large variety of handwriting (currently only support English).
It's a NN that was trained on ~100k different handwriting examples, and it's pretty robust to cursive / sloppy handwriting. Haven't tried European vs American numbers yet but I'll definitely give it a try now that you've piqued my curiosity
The algorithm works on a large variety of handwriting (currently only support English).
It's a NN that was trained on ~100k different handwriting examples, and it's pretty robust to cursive / sloppy handwriting. Haven't tried European vs American numbers yet but I'll definitely give it a try now that you've piqued my curiosity
Thanks for the reply!
The European 1 often looks like an upward facing triangle without a bottom line. Surprisingly frequently it has caused European's in the US to have issues with the IRS/Taxes because the European 1 is often interpreted by Americans as a 7 (because Americans do not use a bar across the center of the 7 or Z). A household size of 7 instead of 1 can be a real back breaker!
Pretty cool project / release! How does it do with bubbly teenage girl script? Example: http://www.annakoren.com/images/love7.gif Best to ya.
thanks for the question!
It handles that kind of script pretty well, mainly because we had a good amount of training examples from college student handwriting
It handles that kind of script pretty well, mainly because we had a good amount of training examples from college student handwriting
>European numbers vs American numbers (the 1 especially)?
Honestly I think 9 will be the bigger issue, to my eyes the European version is near identical to lower-case G.
Honestly I think 9 will be the bigger issue, to my eyes the European version is near identical to lower-case G.
You mean q? Because g has that loop at the end.
Both, really. Check out this small sample from MNIST
https://www.researchgate.net/profile/Amaury_Lendasse/publica...
https://www.researchgate.net/profile/Amaury_Lendasse/publica...
No, I mean G precisely because it has that loop at the bottom. I see it near universally in Poland, and 100% in Germany but with a terribly small sample size.
Great MVP pitch!
After signing up for the Beta, it hit me. Every single bit of this could have been staged without having done any coding at all!! Brilliant!!
I'm not sure if that is actually the case, but theoretically it is totally possible. This is a great example of the classic MVP pitch: validate interest before building.
After signing up for the Beta, it hit me. Every single bit of this could have been staged without having done any coding at all!! Brilliant!!
I'm not sure if that is actually the case, but theoretically it is totally possible. This is a great example of the classic MVP pitch: validate interest before building.
Well, at least part of it was staged. The handwriting on the left differs from the Google Doc:
- Handwriting: "The plan is simple:"
- Google Doc: "The plan is simple and brilliant. Here are the steps:"
https://youtu.be/6bhoj1wr30Y?t=2m13s
- Handwriting: "The plan is simple:"
- Google Doc: "The plan is simple and brilliant. Here are the steps:"
https://youtu.be/6bhoj1wr30Y?t=2m13s
Yep, that's fair.
We staged the google doc bc there is still a 10-30 sec lag time in sending the note out, we'll fix it up. Thank you for the call out!
We staged the google doc bc there is still a 10-30 sec lag time in sending the note out, we'll fix it up. Thank you for the call out!
Honestly, this wasn't calling you out at all. I think it is great. It is a very well done presentation. You did a good enough job that it made me interested in the product.
Thanks Brad! It was a great call out and something we should've seen before posting. Appreciate the feedback
So this is a con? 'staging' is a nice way of putting it I guess.
Hey! We wanted to show how the product works if notes are delivered instantly - vs. the current 10-30 sec lag time we experience between note being snapped, and the transcribed copy you receive.
Definitely didn’t mean for it to put a damper on the tech - sorry for the mistake.
Definitely didn’t mean for it to put a damper on the tech - sorry for the mistake.
Nice good catch!
I totally think this was all staged. I would love to think that they've made some huge innovation in OCR.
In any case, it is a good example of an MVP.
I totally think this was all staged. I would love to think that they've made some huge innovation in OCR.
In any case, it is a good example of an MVP.
Thanks for the catch! We uploaded an updated version of the video here
https://youtu.be/gYPWjYDsRyk
https://youtu.be/gYPWjYDsRyk
Does the processing happen locally or is there some cloud processing going on? Kind of worried about the privacy implications of that.
That's a great question.
It currently happens in the cloud (purely a software design decision), but we could probably do it locally as well since the NN is already trained.
We were thinking of having similar TOS as normal note editing software (Evernote, Dropbox Paper) to mitigate security concerns. What do you think?
It currently happens in the cloud (purely a software design decision), but we could probably do it locally as well since the NN is already trained.
We were thinking of having similar TOS as normal note editing software (Evernote, Dropbox Paper) to mitigate security concerns. What do you think?
I think that although handwriting won't be going away for some time, I've come to find that I'm using digital mediums for note-taking more often and it's becoming more popular (see Apple Pencil, Surface Pro Pen). If this service were coupled to an app that supported writing with a digital pen, it would be more appealing to many; it would mitigate the need to snap a photo, mitigate the need to have paper, and still allow for someone who wants to jot out their ideas by hand to have a digital text copy in the end. Any ideas for going down this route?
I mean, taking photos and rendering paper notes to digital has definitely been done multiple times already. This other approach would become a must-buy app for anyone who uses their ipad pro or surface pro for notes.
I mean, taking photos and rendering paper notes to digital has definitely been done multiple times already. This other approach would become a must-buy app for anyone who uses their ipad pro or surface pro for notes.
Thanks for commenting!
Adding digital handwriting support is a great idea. I actually think other companies do it pretty well, which is why we didn't go down that route. The reason is that they use a different type of algorithm that learns, in part, from the handwriting velocity, and gives you edit access on the go, which is not possible if you've taken the notes in a normal notebook.
We decided to start with plain notebook text mainly because it seemed like no one else had solved this problem to our satisfaction yet.
Adding digital handwriting support is a great idea. I actually think other companies do it pretty well, which is why we didn't go down that route. The reason is that they use a different type of algorithm that learns, in part, from the handwriting velocity, and gives you edit access on the go, which is not possible if you've taken the notes in a normal notebook.
We decided to start with plain notebook text mainly because it seemed like no one else had solved this problem to our satisfaction yet.
No.
I'm working for a company that offers DMS and OCR services.
A big business atm are cheques. I .. don't understand what they're for, consider them weird. But there is a huuge number of places that use them, the US is a part of that for some reason.
A lot of those are filled in by hand.
Say, you're doing a census project. You send out forms that WILL be filled in by hand.
That said, I don't believe in silver bullets here...
A big business atm are cheques. I .. don't understand what they're for, consider them weird. But there is a huuge number of places that use them, the US is a part of that for some reason.
A lot of those are filled in by hand.
Say, you're doing a census project. You send out forms that WILL be filled in by hand.
That said, I don't believe in silver bullets here...
Love the idea and really wish you already supported other languages!
By the way, you have a problem with your demo video: around the 2:10 mark of the video you can see that the first phrase of the .docx file has a lot more content than the written note... While the note contains: 'The plan is simple', the document contains: 'The plan is simple and brilliant. Here are the steps'.
I'm not questioning your tech, but if your service isn't really running on the demo, maybe you could make this explicit somewhere in the video?
By the way, you have a problem with your demo video: around the 2:10 mark of the video you can see that the first phrase of the .docx file has a lot more content than the written note... While the note contains: 'The plan is simple', the document contains: 'The plan is simple and brilliant. Here are the steps'.
I'm not questioning your tech, but if your service isn't really running on the demo, maybe you could make this explicit somewhere in the video?
Good call out, we'll edit the video description
I've been keeping a physical dev diary and have been trying to keep up with transcribing it to digital, but would love any shortcuts to that process as I'm pretty bad about keeping up with it.
My handwriting is kind of messy but I'm eager see how well your algorithms can handle it. It doesn't have to be perfect anyway, as I don't mind going in and cleaning up afterwards. Should still save me some time and some typing.
My handwriting is kind of messy but I'm eager see how well your algorithms can handle it. It doesn't have to be perfect anyway, as I don't mind going in and cleaning up afterwards. Should still save me some time and some typing.
Thanks for the comment! We had the same use case when we decided to build it.
It's currently not perfect but handles a surprising amount of bizarre handwriting styles (cursive / messy notes). Looking forward to hearing your thoughts as we onboard to the beta,
It's currently not perfect but handles a surprising amount of bizarre handwriting styles (cursive / messy notes). Looking forward to hearing your thoughts as we onboard to the beta,
Nice! Maybe I won't need to implement this myself then.
Now if only someone would release designs for an affordable, reliable, non-destructive robot to do the physical data collection... My backlog of notes is way to big to stand around snapping cell phone photos at all of it manually.
Now if only someone would release designs for an affordable, reliable, non-destructive robot to do the physical data collection... My backlog of notes is way to big to stand around snapping cell phone photos at all of it manually.
Or a service that sends you a box where you could use your records, send it to them to digitalize.
Care to share some details on your technology? On whose handwriting was this trained, did you use any public datasets for this? And of course, how well will this perform on writing styles it hasn't seen before?
We did two things to train it (1) scraped the web for photographs of handwritten notes with known transcription to build our training dataset (2) had our university friends / students write out training examples by hand to get more realistic data on what modern handwriting looks like
Scribble currently only supports English, so it does poorly with other languages, but is pretty robust to poor handwriting in English (such as my own).
It gets about 85% of my handwriting correct (my handwriting is abysmal), so there's definitely room for improvement.
Scribble currently only supports English, so it does poorly with other languages, but is pretty robust to poor handwriting in English (such as my own).
It gets about 85% of my handwriting correct (my handwriting is abysmal), so there's definitely room for improvement.
Looks nice. Any chance for an api so we can send jpg and get back a document?
Thank you for the comment!
We don't have a developer facing API at the moment but it's in the roadmap. Once our algorithm is accurate enough that it "just works" in an enterprise setting, we may open up an API so developers can build applications for their businesses.
We don't have a developer facing API at the moment but it's in the roadmap. Once our algorithm is accurate enough that it "just works" in an enterprise setting, we may open up an API so developers can build applications for their businesses.
Is your approach word-based or character based ?
Can't wait to see the final product ! The best neural-based handwriting OCRs are <90% right now, it's good to see something new in the landscape.
Can't wait to see the final product ! The best neural-based handwriting OCRs are <90% right now, it's good to see something new in the landscape.
Thank you for the question!
Currently, it's a combination of the two, mainly because people often take notes hastily so word-based recognition coupled with spell check allow you to fix things on the fly. However, this also results in bizarre outputs sometimes so we're still figuring out what an optimal output looks like.
Currently, it's a combination of the two, mainly because people often take notes hastily so word-based recognition coupled with spell check allow you to fix things on the fly. However, this also results in bizarre outputs sometimes so we're still figuring out what an optimal output looks like.
Cool idea. Could see this being useful with something like Evernote. It's got a great note capture feature as well but doesn't covert to text you can't edit.
Thanks!
Agreed re: Evernote. I actually really like that feature, because it makes handwritten notes searchable but found the same problem you identified with the lack of transcription.
My hope is we can integrate with players like Evernote / OneNote who already do a great job at centralizing notes.
Agreed re: Evernote. I actually really like that feature, because it makes handwritten notes searchable but found the same problem you identified with the lack of transcription.
My hope is we can integrate with players like Evernote / OneNote who already do a great job at centralizing notes.
Can you elaborate on how does it improve on traditional OCR?
Other than that, pretty neat that it keeps (or even improves) on the existing formatting on the note.
Definitely thanks for asking!
Traditional OCRs are good at transcribing typed notes (e.g. pdfs) to editable docs, but do poorly with handwriting. The best OCRs I've seen can make a handwritten note searchable (e.g. Evernote) but still don't transcribe it editable form.
A lot of academic work on transcribing images of handwritten notes into text has surfaced over the last couple of years (mostly regarding using neural networks), and we decided to apply it
Traditional OCRs are good at transcribing typed notes (e.g. pdfs) to editable docs, but do poorly with handwriting. The best OCRs I've seen can make a handwritten note searchable (e.g. Evernote) but still don't transcribe it editable form.
A lot of academic work on transcribing images of handwritten notes into text has surfaced over the last couple of years (mostly regarding using neural networks), and we decided to apply it
This project is really cool. I really do hope you guys have implemented a usable OCR. Has the team looked into integrating with EHR systems?
Thanks!
I was at LensCrafters the other day and had to fill out a paper form that someone input into a computer by hand, so definitely see the need there.
Our goal is to get a high enough accuracy for handwriting OCR to work in enterprise settings. 90% may be good for consumers, but I wouldn't want to put anyone's health on the line due to a transcription error
I was at LensCrafters the other day and had to fill out a paper form that someone input into a computer by hand, so definitely see the need there.
Our goal is to get a high enough accuracy for handwriting OCR to work in enterprise settings. 90% may be good for consumers, but I wouldn't want to put anyone's health on the line due to a transcription error
do you plan to provide an API?
Thank you for the comment!
We don't have a developer facing API at the moment but it's in the roadmap. Once our algorithm is accurate enough that it "just works" in an enterprise setting, we may open up an API so developers can build applications for their businesses.
We don't have a developer facing API at the moment but it's in the roadmap. Once our algorithm is accurate enough that it "just works" in an enterprise setting, we may open up an API so developers can build applications for their businesses.
What about hand drawn figures?
Scribble doesn't transcribe hand drawn figures yet.
I used to be a physics student and always had to draw plots / equations by hand, so it's definitely a feature I would love to see as well
I used to be a physics student and always had to draw plots / equations by hand, so it's definitely a feature I would love to see as well
This could be awesome when combined with Rocketbook
this kinda thing was well studied and developed nearly 30 years ago already.....
Deep leaning, ML are all hypes.
Thank you for commenting!
ML / NN have been around for a while, but there are a few reasons Scribble is only possible now:
1) Although classifying MNIST digits is the "hello world" of ML, doing the same with notes is substantially more difficult. The algorithm has to figure out sentence structure, punctuation, paragraph breaks, lists, and tons of other features that are hard to train. This problem is still a major research topic academically.
2) As a corollary to (1), while OCR has been around for a while, handwriting OCR has never worked due to (1).
3) Computing power has never been so cheap, training the algorithm would have been very expensive before AWS / Azure / etc abstracted hardware and made it inexpensive
ML / NN have been around for a while, but there are a few reasons Scribble is only possible now:
1) Although classifying MNIST digits is the "hello world" of ML, doing the same with notes is substantially more difficult. The algorithm has to figure out sentence structure, punctuation, paragraph breaks, lists, and tons of other features that are hard to train. This problem is still a major research topic academically.
2) As a corollary to (1), while OCR has been around for a while, handwriting OCR has never worked due to (1).
3) Computing power has never been so cheap, training the algorithm would have been very expensive before AWS / Azure / etc abstracted hardware and made it inexpensive
Without any links or proof, this is a low value comment.
Do you need links or proofs if I tell you the sky is blue ?
OCR has been around for decades. This approach is only interesting if it improves OCR via NN. Would love to learn more about this process (the improvement, not the training...).
Cursive?
European numbers vs American numbers (the 1 especially)?
Doctor (or other badly rendered hand writing)?
Seems like this technology isn't really all that useful if it doesn't work on various inputs, especially 'unclean' or 'sloppy' inputs.