Mozilla's Speech-to-Text Engine Is at Risk Following Layoffs(phoronix.com)
phoronix.com
Mozilla's Speech-to-Text Engine Is at Risk Following Layoffs
https://www.phoronix.com/scan.php?page=news_item&px=DeepSpeech-At-Risk
74 comments
I don't understand what Mozilla is. They re-upped the Google deal to the tune of $400-450million/year for the next 3 years, so their revenue is secure. That's great. I can also understand that it makes sense to cut some side-projects that are probably a distraction (and this one may very well fit that category). But if you cut funding to things like MDN, Servo/Rust and move away from Firefox as a core focus (the thing that just brought you another $1.2 billion) .. what the heck is Mozilla now?
Wait, move away from firefox as a core focus ?
Yeah. From https://blog.mozilla.org/blog/2020/08/11/changing-world-chan...
They list their 5 areas to focus on .. and Firefox isn't mentioned by name anywhere. For example:
"New focus on product. Mozilla must be a world-class, modern, multi-product internet organization. That means diverse, representative, focused on people outside of our walls, solving problems, building new products, engaging with users and doing the magic of mixing tech with our values" (emphasis mine)
They list their 5 areas to focus on .. and Firefox isn't mentioned by name anywhere. For example:
"New focus on product. Mozilla must be a world-class, modern, multi-product internet organization. That means diverse, representative, focused on people outside of our walls, solving problems, building new products, engaging with users and doing the magic of mixing tech with our values" (emphasis mine)
The second sentence of the article you linked is:
> Mozilla exists so the internet can help the world collectively meet the range of challenges a moment like this presents. Firefox is a part of this.
They follow up on this by saying that they need to go BEYOND the browser but it doesn't mean that they want to drop it.
But well...you never know with that hollow techmarketing talk...
> Mozilla exists so the internet can help the world collectively meet the range of challenges a moment like this presents. Firefox is a part of this.
They follow up on this by saying that they need to go BEYOND the browser but it doesn't mean that they want to drop it.
But well...you never know with that hollow techmarketing talk...
>but it doesn't mean that they want to drop it.
I wasn't trying to suggest there was no mention of Firefox in their roadmap blog - though it is telling that the Firefox mention you pointed to is in context of an argument for developing non-Firefox products. Nor was I arguing that they will drop Firefox. Even if they wanted to, they wouldn't be able to because Firefox is their sole revenue driver and the only thing that gives them a seat at the table for developing open-standards.
To me the telling part was when they listed their five areas of focus, and Firefox was not one of them.
I wasn't trying to suggest there was no mention of Firefox in their roadmap blog - though it is telling that the Firefox mention you pointed to is in context of an argument for developing non-Firefox products. Nor was I arguing that they will drop Firefox. Even if they wanted to, they wouldn't be able to because Firefox is their sole revenue driver and the only thing that gives them a seat at the table for developing open-standards.
To me the telling part was when they listed their five areas of focus, and Firefox was not one of them.
Firefox is getting its ass beat in the market right now. It's essentially a dying product.
I'm seing more Firefox than before in my stats, not less. Not a lot more, but a few %.
"Not dead yet"
"Not dead yet"
I personally have the feeling that their main focus is on promoting wokeness/social justice in the software industry. They probably claim that their core competency is still browser technology, but I am not so sure when looking at our code and seeing comments like this:
// Change these to proper private fields once Firefox supports themThe phrase "get woke go broke" comes to mind unfortunatly.
I'm a heavy Firefox user, and have given money to Mozilla in the past, but their leadership has dropped the ball and I'm sorry to say I'm reluctant to give them any more $.
My hope is that they focus efforts on Firefox as an open-web, security and privacy-promoting system (which it does a lot of already). I do think things like Rust have been a distraction, the world doesn't need yet another programming language (compared to Modern C++, Rust doesn't seem worth the effort?).
I'm a heavy Firefox user, and have given money to Mozilla in the past, but their leadership has dropped the ball and I'm sorry to say I'm reluctant to give them any more $.
My hope is that they focus efforts on Firefox as an open-web, security and privacy-promoting system (which it does a lot of already). I do think things like Rust have been a distraction, the world doesn't need yet another programming language (compared to Modern C++, Rust doesn't seem worth the effort?).
> the world doesn't need yet another programming language
I disagree there. I think the world needed a systems-level language to replace C++ with safer constructs, more consistent syntax and no historical baggage. It's an open question whether Mozilla was the right sponsor for the development of that language, but nobody else was there (Sorry Dlang =) ).
I disagree there. I think the world needed a systems-level language to replace C++ with safer constructs, more consistent syntax and no historical baggage. It's an open question whether Mozilla was the right sponsor for the development of that language, but nobody else was there (Sorry Dlang =) ).
> Mozilla was the right sponsor for the development of that language, but nobody else was there.
Which may indicate that the world really doesn't need another programming language.
Which may indicate that the world really doesn't need another programming language.
I take your point and you may be right. I don't do a lot of C++ development, so I can't speak from first-hand experience if Modern C++ makes Rust superfluous.
The borrow checker alone is worth the price of admission for Rust, you can't do anything quite like that in C++.
Plucking a single comment entirely out of context and presenting it as damning evidence against an entire company's competency feels kind of over the top.
The context is that Firefox is not supporting essential OOP features (like private fields) in Javascript, which Chrome has been supporting since April 2019. In the past, Firefox had a lot of experimental features and was often the first browser to support new stuff. This has changed, Firefox is now behind in quite a few areas and if you look at their priorities, management doesn't really seem to care.
> The context is that Firefox is not supporting essential OOP features (like private fields)
I'd dispute private fields being "essential". People have been writing a lot of OOP code in JavaScript without them for a long time now.
I'd dispute private fields being "essential". People have been writing a lot of OOP code in JavaScript without them for a long time now.
I’m glad to see the healthy skepticism towards Mozilla lately, they have long been a sacred cow in tech.
Just because a company’s mission is positive or aligns with ones values does not mean the company itself is good or is actually carrying that mission out.
Just because a company’s mission is positive or aligns with ones values does not mean the company itself is good or is actually carrying that mission out.
Agree, but you seem to be turning downvoted-transparent.
I am also always surprised that Mozilla receives so much praise for its technology stack, when their core source code for Firefox and Thunderbird is such a buggy pain to work with.
I understand that they inherited a legacy codebase, but I'm not sure that playing catch-up with Google on (in my opinion) useless features like WebUSB was good technical management. In fact, I would argue that letting technical debt accumulate to the point where you need to rewrite large parts in a completely different language and deprecate technologies still in use (e.g. XUL) is more like an admission of failed technical leadership.
So I am honestly surprised that Mozilla is cutting projects and firing developers, yet nobody seems to take action to reduce the overhead of their apparently incapable management.
I am also always surprised that Mozilla receives so much praise for its technology stack, when their core source code for Firefox and Thunderbird is such a buggy pain to work with.
I understand that they inherited a legacy codebase, but I'm not sure that playing catch-up with Google on (in my opinion) useless features like WebUSB was good technical management. In fact, I would argue that letting technical debt accumulate to the point where you need to rewrite large parts in a completely different language and deprecate technologies still in use (e.g. XUL) is more like an admission of failed technical leadership.
So I am honestly surprised that Mozilla is cutting projects and firing developers, yet nobody seems to take action to reduce the overhead of their apparently incapable management.
Maybe I'm unlucky, but any company I worked in has that legacy codebase which kind-of works, has bugs, and that nobody wants to refactor - and I don't blame them, refactoring code which had years of debugging is not always a good idea if you don't have unlimited money.
I also wouldn't blame them for not fixing old bugs if they had a shortage of developers. But letting critical security bugs go un-patched, so that your team can work on obscure shiny new stuff instead, that looks like wrong priorities to me.
Nobody would switch to Firefox for WebUSB, which is why I chose it as the example. There's only crickets on Twitter: https://twitter.com/hashtag/webusb
Instead, people switch to Firefox for privacy, safety and reliability. Mozilla should prioritize those areas highly. But in the past, they haven't. Instead, they burned a lot of money on acquiring 3rd party services that many Firefox users see as nothing more than unnecessary bloat. Like $30 mio for Pocket.
That's why I point the blame at their management, not their code base or developers.
And just FYI, I'm pretty sure I could hire a world-class team and build an excellent Speech-to-Text Engine just from Mozilla's management budget for a single year. Because AFAIK, their (in my opinion incapable) management pocketed $80 mio in 2018.
https://assets.mozilla.net/annualreport/2018/mozilla-fdn-201...
Nobody would switch to Firefox for WebUSB, which is why I chose it as the example. There's only crickets on Twitter: https://twitter.com/hashtag/webusb
Instead, people switch to Firefox for privacy, safety and reliability. Mozilla should prioritize those areas highly. But in the past, they haven't. Instead, they burned a lot of money on acquiring 3rd party services that many Firefox users see as nothing more than unnecessary bloat. Like $30 mio for Pocket.
That's why I point the blame at their management, not their code base or developers.
And just FYI, I'm pretty sure I could hire a world-class team and build an excellent Speech-to-Text Engine just from Mozilla's management budget for a single year. Because AFAIK, their (in my opinion incapable) management pocketed $80 mio in 2018.
https://assets.mozilla.net/annualreport/2018/mozilla-fdn-201...
Mozilla senior leadership have also been caught up in the politically correct mantra that pervades much of silicon valley, which imho is a huge distraction from making great products. Case in point: https://mobile.twitter.com/newstephen/status/106048929137627...
Partial transcript:
"We aspire to the idea that a human being can demonstrate competence and earn respect and leadership and authority based on what you do and how you do it."
Video was posted on Nov 8 2018. I think everyone can draw their own conclusions.
"We aspire to the idea that a human being can demonstrate competence and earn respect and leadership and authority based on what you do and how you do it."
Video was posted on Nov 8 2018. I think everyone can draw their own conclusions.
This has to take the cake for one of the stupidest "we won't use this word because it's offensive or whatever" arguments out there.
Just because people call things that aren't a meritocracy a meritocracy does not mean that's what it means. It's like saying we don't want to say "democracy" anymore because China (PRC) calls itself a democracy and they actually aren't. Or if we turn the whole thing around, saying that "feminism" is offensive because TERFs use the term too.
Like, are there actually people who actually think like that? How do they communicate??
Just because people call things that aren't a meritocracy a meritocracy does not mean that's what it means. It's like saying we don't want to say "democracy" anymore because China (PRC) calls itself a democracy and they actually aren't. Or if we turn the whole thing around, saying that "feminism" is offensive because TERFs use the term too.
Like, are there actually people who actually think like that? How do they communicate??
The word was actually coined for the purposes of mocking the concept of a 'meritocracy' in a satirical essay. Then people picked it up and started using it straight. Now it's come full circle again.
https://en.m.wikipedia.org/wiki/Meritocracy#Etymology
https://en.m.wikipedia.org/wiki/Meritocracy#Etymology
Utopia also was created to mock people seeking to create a perfect world. Nowadays people are trying to create a socialist utopia.
The meaning of words is constantly perverted
The meaning of words is constantly perverted
Don't put a lawyer in charge of a tech company.
"But I think most would also agree that if all corporate lawyers, bank lobbyists, or marketing gurus were to similarly vanish in a puff of smoke, the world would be at least a little bit more bearable."
I remember just yesterday there was a great article on HN that discussed bullshit jobs and counted corporate lawyers towards the "goons". The rationale was that if nobody has them, nobody else needs them.
https://www.businessinsider.com.au/bullshit-jobs-australia-2...
I remember just yesterday there was a great article on HN that discussed bullshit jobs and counted corporate lawyers towards the "goons". The rationale was that if nobody has them, nobody else needs them.
https://www.businessinsider.com.au/bullshit-jobs-australia-2...
Who exists to enforce the rules? Contracts? Etc. I feel like this sounds great on the surface, but is quite flawed.
Agreed. Lawyers (and by extension, the legal system) must exist to ensure order and keep the system fair for as many people as possible. They don’t always achieve that goal, but removing them entirely would make things 100x worse. My favorite example is the ACLU.
It's kind of like the military in that sense. Anything 'PvP' falls in that category.
You can put them in charge of a country, as long as they are effective.
"Hate the game, not the player", might be appropriate here, but even in corporate games players can simply be so ineffective that they should be replaced by someone who can actually play the game.
"Hate the game, not the player", might be appropriate here, but even in corporate games players can simply be so ineffective that they should be replaced by someone who can actually play the game.
Semi-related, but does anyone know if Firefox Send is on the chopping block? They took it down before the layoffs due to security concerns, and now I'm worried it won't be coming back online.
Every alternative I've been able to find has been either poor quality or too shady for me to ever actually want to use.
Every alternative I've been able to find has been either poor quality or too shady for me to ever actually want to use.
Not given priority right now - it's getting abused by malware authors. Confirmed via Mozilla IRC.
You can set it up yourself really easily (got it running locally on Mac in 5 mins), throw an auth'd gateway in front of it to keep it away from bad actors: https://github.com/mozilla/send
You can set it up yourself really easily (got it running locally on Mac in 5 mins), throw an auth'd gateway in front of it to keep it away from bad actors: https://github.com/mozilla/send
I've mostly heard of people using Magic Wormhole for this https://github.com/warner/magic-wormhole Never used it myself
That works great for sending files between my machines or to other technically minded people, but Firefox send was useful to send to others.
I've tried many of the options, and Croc [0] has always worked best for me. I tried Firefox Send, but the downsides were that the entire file had to finish uploading first and a lack of auto-resume.
[0] https://github.com/schollz/croc
>Every alternative I've been able to find has been either poor quality or too shady for me to ever actually want to use.
have you tried https://catbox.moe/ (or others https://archiveteam.org/index.php?title=Pomf.se/Clones)? Not sure what you mean by "shady", but most of them provide a direct download without any interstitials.
have you tried https://catbox.moe/ (or others https://archiveteam.org/index.php?title=Pomf.se/Clones)? Not sure what you mean by "shady", but most of them provide a direct download without any interstitials.
I just started renting my own cloud storage space for moments like this.
There's https://file.re/ that's worked okay for me.
My quick test of DeepSpeech, as posted on Facebook back in April:
--
I just ran a speech-to-text converter on a very clear clip of former Doctor Who actor Tom Baker talking in an interview.
The DeepSpeech converter uses the very latest AI deep-learning advancements to 'listen' to the audio and output the spoken words as text.
After 3 long minutes of running it on a 30-second clip, it printed out its interpretation:
"hooloomooloo how booboorowie i have a honeymoon"
--
I just ran a speech-to-text converter on a very clear clip of former Doctor Who actor Tom Baker talking in an interview.
The DeepSpeech converter uses the very latest AI deep-learning advancements to 'listen' to the audio and output the spoken words as text.
After 3 long minutes of running it on a 30-second clip, it printed out its interpretation:
"hooloomooloo how booboorowie i have a honeymoon"
It's supposed to work on sentence long audio (4 - 5 seconds), they suggest chunking your audio first: https://discourse.mozilla.org/t/longer-audio-files-with-deep...
Also there's essentially two parts to this, the neural net is used for speech-to-characters, and then a language model is used to convert the character stream to words.
I found that the language model they supplied was trained data that did not contain the words I needed, and got significantly improved results when making my own language model using the kenlm[1] tools.
[1]: https://kheafield.com/code/kenlm/
I found that the language model they supplied was trained data that did not contain the words I needed, and got significantly improved results when making my own language model using the kenlm[1] tools.
[1]: https://kheafield.com/code/kenlm/
Would it be possible to substitute this for GPT2/BERT? Or is that a different type of language model? Can the pre-trained language model be fine-tuned? I’m using DeepSpeech to transcribe long-form lecture audio, and have just assumed there would be a massive improvement once they noise-harden the models with 1.0.
GPT2 is not a good language model but there are things like XLM. Mozilla DeepSpeech doesn't support XLM rescoring, other toolkits do and it gives great improvement in accuracy. If you care about accurate transcription you'd better consider alternatives.
I didn't know any other ML-based open source transcription engines existed? I can't seem to find them on Google.
I didn’t know Tom Baker was Welsh.
When I tried it out with English and German phone recordings, it was working competitively. I would have ranked it better than Amazon but worse than Google.
Did you maybe not convert your WAV to the correct sampling rate?
Did you maybe not convert your WAV to the correct sampling rate?
Experiment is worthless for drawing conclusions if it's not reproducible by other people.
Besides the hypothesis that DS sucks, the software could also very well be just fine and you made methodological errors.
Besides the hypothesis that DS sucks, the software could also very well be just fine and you made methodological errors.
It's always sad to see people at the risk of losing their job or the project they are working on. That being said, this Speech to Text engine always seemed to be more a distraction than anything else. There is of course a lot of pessimism at the moment toward Mozilla, and I subscribe to it for the most part, but if the recent changes and upcoming changes result in more focused projects and efforts that wouldn't necessarily be a bad thing.
Not a big problem given so many alternatives around like
E.g. some very active projects are:
* Kaldi (https://github.com/kaldi-asr/kaldi/) obviously, probably the most famous one, and most mature one. For standard hybrid NN-HMM models and also all their more recent lattice-free MMI (LF-MMI) models / training procedure. This is also heavily used in industry (not just research).
* ESPnet (https://github.com/espnet/espnet), for all kind of end-to-end models, like CTC, attention-based encoder-decoder (including Transformer), and transducer models.
* Espresso (https://github.com/freewym/espresso).
* Google Lingvo (https://github.com/tensorflow/lingvo). This is the open source release of Googles internal ASR system, and used by Google in production (their internal version of it, which is not too much different).
* NVIDIA OpenSeq2Seq (https://github.com/NVIDIA/OpenSeq2Seq).
* Facebook Fairseq (https://github.com/pytorch/fairseq). Attention-based encoder-decoder models mostly.
* Facebook wav2letter (https://github.com/facebookresearch/wav2letter). ASG model/training.
* Vosk (https://github.com/alphacep/vosk-api). Offline lightweight speech recognition API with support for 10 languages.
And there are much more.
E.g. some very active projects are:
* Kaldi (https://github.com/kaldi-asr/kaldi/) obviously, probably the most famous one, and most mature one. For standard hybrid NN-HMM models and also all their more recent lattice-free MMI (LF-MMI) models / training procedure. This is also heavily used in industry (not just research).
* ESPnet (https://github.com/espnet/espnet), for all kind of end-to-end models, like CTC, attention-based encoder-decoder (including Transformer), and transducer models.
* Espresso (https://github.com/freewym/espresso).
* Google Lingvo (https://github.com/tensorflow/lingvo). This is the open source release of Googles internal ASR system, and used by Google in production (their internal version of it, which is not too much different).
* NVIDIA OpenSeq2Seq (https://github.com/NVIDIA/OpenSeq2Seq).
* Facebook Fairseq (https://github.com/pytorch/fairseq). Attention-based encoder-decoder models mostly.
* Facebook wav2letter (https://github.com/facebookresearch/wav2letter). ASG model/training.
* Vosk (https://github.com/alphacep/vosk-api). Offline lightweight speech recognition API with support for 10 languages.
And there are much more.
Am I the only one trying to figure out why Mozilla even has a "speech-to-text" team?
They want to build technologies which bring access for everybody. There is a huge population of people who can't type easily and where speech is an important alternative. Having all input running through an ad company (Google) or requiring to buy premium products (Apple) isn't good for an inclusive society valuing privacy.
Quoting their mission statement: "Our mission is to ensure the Internet is a global public resource, open and accessible to all. An Internet that truly puts people first, where individuals can shape their own experience and are empowered, safe and independent." https://www.mozilla.org/en-US/mission/
However Mozilla seems like it did too many wrong bets and didn't execute well (see also FirefoxOS)
Quoting their mission statement: "Our mission is to ensure the Internet is a global public resource, open and accessible to all. An Internet that truly puts people first, where individuals can shape their own experience and are empowered, safe and independent." https://www.mozilla.org/en-US/mission/
However Mozilla seems like it did too many wrong bets and didn't execute well (see also FirefoxOS)
The world needs a state of the art and easy to integrate library for providing speech to text so that we don't have to rely on sending all of our voice data to the big tech giants. I imagine this was the motivation behind it.
That being said, I think the current open source options are actually okay and constantly getting better. They're certainly a lot better today than they were 3 years ago.
That being said, I think the current open source options are actually okay and constantly getting better. They're certainly a lot better today than they were 3 years ago.
The thing is, good speech to text AI needs good training data. And lots of it. With as little error as possible.
They were also doing this. And made an easy to use website, where you could contribute and people did. I cannot imagine it was soo expensive that they now have to throw it under the bus.
They were also doing this. And made an easy to use website, where you could contribute and people did. I cannot imagine it was soo expensive that they now have to throw it under the bus.
I agree with you, their common voice project is extremely important. From the outside it seems like the engineering effort behind this is done -- They just need to keep the site running to keep collecting data. Actually, now that I think about it, validating the data might be a lot of man hours.
I very much hope they don't abandon it, the common voice project is arguably more important than their speech to text engine. There are competing open source speech to text engines. There is no other project like common voice that I am aware of.
I very much hope they don't abandon it, the common voice project is arguably more important than their speech to text engine. There are competing open source speech to text engines. There is no other project like common voice that I am aware of.
"validating the data might be a lot of man hours."
It totally is and therefore also a very good crowdsourced project. And they set up already a nice website with badges and other things motivating people to contribute. Daily 15 minutes of a lot of people would mean lots of validated data. Because with user accounts you can also validate userquality etc.
It totally is and therefore also a very good crowdsourced project. And they set up already a nice website with badges and other things motivating people to contribute. Daily 15 minutes of a lot of people would mean lots of validated data. Because with user accounts you can also validate userquality etc.
You've actually touched on something very important here: Mozilla seems to be getting its fingers into more than it can handle. It wants to do everything but ends up doing a lot of things fairly poorly.
I switched away from Firefox to Brave yesterday. Became tired of how Firefox threads would consume 99% of my CPU endlessly until they're manually killed. This happened about four or five times a day, I'd only notice when performance of everything else tanks. I wish Mozilla would direct more resources towards Firefox than Lockwise and all the other half-baked junk they're spending developer time on.
I switched away from Firefox to Brave yesterday. Became tired of how Firefox threads would consume 99% of my CPU endlessly until they're manually killed. This happened about four or five times a day, I'd only notice when performance of everything else tanks. I wish Mozilla would direct more resources towards Firefox than Lockwise and all the other half-baked junk they're spending developer time on.
In my experience Chromium does this too (busy spinning threads).
hmm, Firefox Voice was supposed to eventually use the speech-to-text engine that's now under threat but instead currently uses Google's engine? Can't imagine priorities like this came up in their recent negotiations.
This (text to speech) breaks pulse audio on my machine every time I enable reader view. Even if I'm not using text to speech or listening to anything else, I'll get a whiny noise until I restart pulse.
The browser does have this effect on Linux systems with pulseaudio. You will hear lots of static in your speakers, and after that the audio will be distorted until you restart pulseaudio.
What I cannot confirm is the relation of this to the speech-to-text engine. Reader mode most likely utilizes text to speech.
What I cannot confirm is the relation of this to the speech-to-text engine. Reader mode most likely utilizes text to speech.
foolzcrow(1)
Seems unlikely. The layoffs happened before they got a new deal:
https://www.theverge.com/2020/8/15/21370020/mozilla-google-f...
Now they can bring people back.
Now they can bring people back.
>Now they can bring people back.
Or they can increase executive compensation.
How far in advance do you think these deals are decided before public announcement?
Or they can increase executive compensation.
How far in advance do you think these deals are decided before public announcement?
Anyone who left isn't coming back. They would now go for better opportunities.
You think they didn't know this deal was going to be signed before they announced their layoffs?
Now I'm not the most knowledgeable when it comes to managing companies so I can't be sure if that is the right move or not. However surely any management that announces massive lay-offs to then turn around and hire those same people back a few weeks later would have to be deemed blatantly incompetent (if not malicious)?