With an open system/engine, you can train your own personal speech model. For kaldi-active-grammar (https://github.com/daanzu/kaldi-active-grammar), you can do so without all that much difficulty, although the process/documentation could certainly use improvement.
I bootstrapped my personal speech model by retaining the commands from me using WSR. My voice is quite abnormal, and it took only 10 hours of speech data to train a model orders of magnitude more accurate than any generic model I've ever used. And of course, I retain much of my usage now with Kaldi, so my model improves more and more over time. A virtuous flywheel!
Yep, I have been using my Kaldi backend through Dragonfly exclusively ever since I got v0.1.0 working.
I bootstrapped writing it initially using the Dragonfly WSR (windows speech recognition) backend, because that gave me the best accuracy out of the available options at the time. All of my development of it since the initial working version has been done using each previous version, so now it is basically bootstrapped itself. My productivity skyrocketed once I switched to Kaldi, due to being able to use my custom trained speech model just for my voice for orders of magnitude better accuracy, plus dramatically lower latency. (And it freed me from being dependent on closed software out of my control.)
I bootstrapped my personal speech model by retaining the commands from me using WSR. My voice is quite abnormal, and it took only 10 hours of speech data to train a model dramatically more accurate than any generic model I've ever used. And of course, I retain much of my usage now with Kaldi, so my model improves more and more over time. A virtuous flywheel!
"Everything other than talon has terrible latency": False! I develop kaldi-active-grammar (https://github.com/daanzu/kaldi-active-grammar), a free and open source speech recognition backend, which has extremely low latency. You can adjust how aggressive the VAD (voice activity detection) is to suit your preference, but the speech engine latency can be almost negligible, especially for voice commands (vs prose dictation). However, I agree that "most existing speech recognition engines were not designed with the kind of latency you want for quick one syllable commands", and that low latency is pivotal to being productive with voice commands. I also agree with your other points.
I have been coding entirely by voice for approximately 10 years now (by hand long before that). Most of that time I have been using the Dragonfly (https://github.com/dictation-toolbox/dragonfly) library to construct my own customized voice coding system. The library is highly flexible and open source, allowing you to easily customize everything to suit what you need to be productive. It is perhaps the power user analogue to Dragon Naturally Speaking. With it, you can certainly be highly productive coding by voice. However, it does require work to setup and customize to suit you, so it isn't really for the "general population" of computer users to just sit down and use. With regard to accuracy of speech recognition, being open allows you to (with sufficient motivation) to train a custom acoustic speech model that recognizes your voice specifically extremely well.
Regarding the software packages you referenced: Yes, Dragon is trash that I want nothing to do with, because of its inefficient interface, its complete inability to accurately understand my voice, and its generally shoddy software quality. Voice Computer (which I hadn't seen before) is therefore eliminated as well, though it doesn't look terrible as a front end to Dragon to better use the OS GUI-accessibility info. Many people like Talon, but I demand something open, which I can modify to suit my needs.
I have been coding entirely by voice for approximately 10 years now (by hand long before that). Most of that time I have been using the Dragonfly (https://github.com/dictation-toolbox/dragonfly) library to construct my own customized voice coding system. The library is highly flexible and open source, allowing you to easily customize everything to suit what you need to be productive. It is perhaps the power user analogue to Dragon Naturally Speaking. With it, you can certainly be highly productive coding by voice. In fact, I develop kaldi-active-grammar (https://github.com/daanzu/kaldi-active-grammar), a free and open source speech recognition backend usable by Dragonfly, itself entirely by voice. There's also a community of voice coders using Dragonfly and other tools that build on top of it, such as Caster (https://github.com/dictation-toolbox/Caster).
I want to point out that the Dragonfly library has backends other than the closed/commercial Dragon, and there has been significant progress expanding beyond it recently. I have been developing kaldi-active-grammar [0] for the past couple years as a fully open source, modifiable, cross-platform, and free alternative to Dragon when used with Dragonfly. While KaldiAG misses some of the user-friendly niceties of Dragon, these are least helpful when coding by voice. And KaldiAG has benefits such as lower latency that improves coding by voice tremendously.
I created KaldiAG because I didn't trust relying on closed source software for something so crucial to my productivity, where a decision by an outside party determines whether I can function. As a bonus, open source means I can make it work better to fit my needs than closed source ever could.
You raise good points. For what it's worth, I think all "invalidated" samples are still included in the distribution (invalidated.tsv), with the number of up and down votes for each (but not the reasoning).
Google is certainly doing some great work with this, both Project Euphonia and other research [0]. However, as far as I know, the Euphonia dataset is closed and only usable by Google. A Common Voice disordered speech dataset would (presumably) be open to all, allowing independent projects and research. (I would love to have access to such a dataset.)
Gathering, collecting, and publishing such a dataset would be great, and would certainly much improve the baseline speech recognition for people with disordered speech, but it can only help so much without personalizing to a specific individual. This is true for anybody, but more so for disordered speech. This is an area where I think "generic" solutions will inevitably struggle, even if they are somewhat specialized on "generic disarthritic" speech.
However, this means that the gains to be had from personalized training are greater for disordered speech than for "average" speech. I develop kaldi-active-grammar [0], which specializes the Kaldi speech recognition engine for real-time command & control with many complex grammars. I am also working on making it easier to train personalized speech models, and to fine tune generic models with training for an individual. I have posted basic numbers on some small experiments [1]. Such personalized training can be time consuming (depending on how far one wants to take it), but as my parent comment says, disabled people may need to rely more on ASR, which means they have that much more to gain by investing the time for training.
Nevertheless, a Common Voice disordered speech dataset would be quite helpful, both for research, and for pre-training models that can still be personalized with further training. It is good to see (in my sibling comment) that it is being discussed.
I develop kaldi-active-grammar [0]. The Kaldi engine itself is state of the art and open source, but is focused on research rather than usability. My project has a simple interface and comes with a pretty good open source speech model.
However, kaldi-active-grammar specializes in real time command and control, with advanced features that don't really apply to your use case. Vosk [1] is probably a simpler, better fit for you. It likewise uses Kaldi and can use my models, and offers some others of its own as well.
Neither are particularly focused on transcription per se, but they are open.
Just to be clear, the Dragonfly speech recognition command and control framework has multiple "backends" (speech recognition engines), including my Kaldi one. Probably the most used one currently is the Dragon Naturally Speaking backend.
The Kaldi engine, being developed primarily for research in speech recognition, can support a huge variety of "models". I think the consensus general best for most use cases (particularly for real time, low latency, streaming use) currently would be considered to be the "nnet3 chain" models, which are what my kaldi-active-grammar uses/supports.
For me at least, dictation is actually the more straining mode of speech recognition, as compared to using my command grammars. With dictation, you might say anything, so the computer is given wide leeway in what to recognize, and so you must speak as clearly as possible. With commands (especially a nice simple command grammar), however, what you can say is greatly restricted, which allows you the freedom to speak indistinctly and still be understood by the computer. This can even be magnified by personalized training of the speech recognition model.
When using commands at my computer, I frequently find myself muttering and grunting things that even I think to myself (that is utterly un-understandable), yet the computer understands just fine. Dictating for prolonged periods can be tiring for me, but I can happily code by voice commands all night.
I agree with everything you said, but I would add that a critical component of voice command and control is strict grammars. There is so much structure and context in what we speak, and being able to limit what can be recognized to only what can be reasonably spoken (based on the current context) can allow massive increases in accuracy. (EDIT: ah, you edited to add a mention of this as well.)
And one shameless plug deserves another! Vosk is a great project, but my kaldi-active-grammar [0] (mentioned in another comment here) also uses the same Kaldi engine, but extends it and is designed specifically for this use case. It supports defining many grammars, in any combination, and activating/deactivating them at will instantly per-utterance. I think it's probably a better fit as a backend for your project than vosk. My work focuses on the backend technology, so it would be great to have more front ends using it to put it within users' reach (so to speak).
Windows Speech Recognition is far from the best, so perhaps your trouble could be partly caused by how you had to speak in order to be understood, rather than the command style? I used to use WSR to code by voice, and it was far more laborious than my current setup.
I develop kaldi-active-grammar [0]. The Kaldi engine is state of the art for command and control. Although I don't have the data and resources for training a model like Microsoft/Nuance/Google, being an open rather than closed system allows me to train models that are far more personalized than the large commercial/generic ones you are used to. For example, see the video of me using it [1], where I can speak in a relaxed manner without having to over enunciate and strain my voice.
Gathering the data for such training does take some time, but the results can be huge [2]. Performing the actual training is currently complicated; I am working on making it portable and more turnkey, but it's not ready yet. However, I am running test training for some people. Contact me if you want me to use you as a guinea pig.
I wrote a simple little Python GUI app to record training audio. Given a text file containing prompts, it will choose a random selection and ordering of them, display them to be dictated by the user, and record the dictation audio and metadata to a .wav file and recorder.tsv file respectively. You can select a previous recording to play it back, delete it,
and/or re-record it. It comes with a few selections of sentences designed to cover a broad diverse range of English (Arctic, TIMIT). Pretty simple and no-nonsense.
Originally intended for recording data for training speech recognition models [0], it should work just as well for recording to be used for speech synthesis.
I don't know much about Home Assistant, but that certainly should be possible to set up. The KaldiAG API is pretty low level, but basically: you define a set of rules, and send in audio data, along with a bit mask of which rules are active at the beginning of each utterance, and receive back the recognized rule and text. The easy solution is probably to go through Dragonfly, which makes it easy to define the rules, contexts, and actions. It might be a little hacky to do, but you should be able to wire it up with Home Assistant somehow.
Although I mainly use it for computer control as demonstrated in the video, I do have many commands akin to home automation, like adjusting the lights, HVAC, etc.
Too late to edit, but I should probably have noted that KaldiAG also would make it easy to define "contexts" when (groups of) commands are active for recognition. For example, if the TV is on, you could have commands for adjusting the volume/etc. But if it is off, those commands are disabled, so they can't be recognized, and further, the engine knows this and can therefore better recognize the other commands that remain active.
I develop Kaldi Active Grammar [1], which is mainly intended for use with strict command grammars. Compared to normal language models, these can provide much better accuracy, assuming you can describe (and speak) your command structure exactly. (This is probably more acceptable for a voice assistant for an audience that is more technical.) The grammar can be specified by a FST, or you can use KaldiAG through Dragonfly, which allows you to specify them (and their resultant actions) in Python. However, KaldiAG can also do simple plain dictation if you want.
KaldiAG has an English model available, but other models could be trained. Although you can't just drop in and use a standard Kaldi model with KaldiAG, the modifications required are fairly minimal and don't require any training or modification of its acoustic model. All recognition is performed locally and off line by default, but you can also selectively choose to do some recognition in the cloud, too.
Kaldi generally performs at the state of art. As a hybrid engine, although training can be more complicated, it generally requires far less training data to achieve high accuracy, compared to "end to end" engines.
If you are willing to do some training, you can get tremendously improved results, in my experience. For what it's worth, my voice is quite abnormal, so most untrained speech recognition is terrible for me, and even performing the normal "training" for Dragon still resulted in very poor accuracy. However, apparently their training is quite limited, because once I developed Kaldi Active Grammar [1], and did my own direct training, the results were fantastic in comparison, with orders of magnitude better accuracy. The personalized training is still pretty new and raw, and it needs a lot of setup to do the training itself currently.
https://github.com/daanzu/kaldi-active-grammar