Encoder in the T5 sense doesn't produce a fixed vector, it produces one encoded vector for every step of input and all of that is given to the decoder.
The only difference between encoder/decoder and decoder-only is masking:
In an encoder, none of the tokens are masked at any step, and are all visible in both directions to the encoder. Each output of the encoder can attend to any input of the encoder.
In the decoder, the tokens are masked causally - each N+1 token can only attend to the previous N tokens.
You are missing the speech decoding part. I can't speak to why the clients you were working on were doing what they were doing. For a different reference point see the cloud streaming api.
Possibly confusions from that doc: "RNN-T" is entirely orthogonal to RNNs (and not the only streamable model). Attention is also orthogonal to streaming. A chunked or sliding window attention can stream, a bi-directional RNN cannot. How you think of an encoder and a decoder streaming is also different.
At a practical level, if a model is fast enough, and VAD is doing an adequate job, you can get something that looks like "streaming" which a non-streaming model. If a streaming model has tons of look-ahead or a very large input chunk size, its latency may not feel a lot better.
Where the difference is sharp is where VAD is not adequate: Users speak in continuous streams of audio, they leave in unusual gaps within sentences and run sentences together. A non-streaming system either hurts quality because sentences (or even words) get broken up that shouldn't, or has to wait forever and doesn't get a chance to run, when a streaming system would have already been producing output.
And to your points about echo cancellation and interference: There's many text only operations that benefit from being able to start early in the audio stream, not late.
I just went through process of helping someone stand up an interactive system with whisper etc and the lack of an open sourced whisper-quality streaming system is such a bummer because it really is so much laggier than it has to be.
This is exactly what Google ASR does. Give it a try and watch how the results flow back to you, it certainly is not waiting for VAD segment breaking. I should know.
Streaming used to be something people cared about more. VAD is always part of those systems as well, you want to use it to start segments and to hard cut-off, but it is just the starting off point. It's kind of a big gap (to me) that's missing in available models since Whisper came out, partly I think because it does add to the complexity of using the model, and latency has to be tuned/traded-off with quality.
It is not streaming in the way people normally use this term. It's a fuzzy notion but typically streaming means something encompassing:
- Processing and emitting results on something closer to word by word level
- Allowing partial results while the user is still speaking and mid-segment
- Not relying on an external segmenter to determine the chunking (and therefore also latency) of the output.
Streaming for TTS doesn't matter but for speech to text it is more meaningful in interactive cases. In that case the user's speech is arriving in real time and streaming can mean a couple levels of things:
- Overlap compute with the user speaking: Not having to wait until all the speech has been acquired can massively reduce latency at the end of speech and allow a larger model to be used. This doesn't have to be the whole system, for instance an encoder can run in this fashion along audio as it comes in even if the final step of the system then runs in a non-streaming fashion.
- Produce partial results while the user is speaking: This can be just a UI nice to have, but it can also be much deeper, eg, a system can be activating on words or phrases in the input before the user is finished speaking which can dramatically change latency.
- Better segmentation: Whisper + Silero is just using VAD to make segments for Whisper, this is not at all the best you can do if you are actually decoding while you go. Looking at the results as you go allow you to make much better and faster segmentation decisions.
Unfortunately it's not, this is some supplementary video. The full thing is 44 minutes long and covers a lot of different angles. The ongoing story about the Czech Xenon clinic in it that he covers over a longer time period is especially crazy.
The Xenon episode of Hamilton's Pharmacopeia is one of the very best episodes of that series, I don't know the best way to find it on streaming currently, it was available I believe on Netflix (or maybe it was Hulu?): https://www.vicetv.com/en_us/video/xenon-the-perfect-anesthe...
The full episode takes very disturbing twists and turns, worth a watch.
You asked "why would someone hide prep", you got one example. It is by no means an exhaustive list. For example, a big fear is being perceived as having HIV (since prep drugs are also part of HIV treatment).
If you do actually have interest in this topic you could read about it:
"Men were able to initiate PrEP without discussing it with their partners, whereas some women said they needed to get permission. Discussions around starting PrEP could raise questions about trust and infidelity and act as a barrier to PrEP use."
"the men are forced to confront the fact..." No they aren't, they simply never confront it. People go to their graves denying that they have HIV, denying that they ever tested positive, denying that a positive HIV test has anything to do with illness.
"Otherwise the men are setting themselves up for a lose:lose scenario regardless of what the wife does." - Of course.
I mean, the most rational win:win thing to do is to get an HIV test and get treated if positive. They then both don't get sick and die and can't pass along HIV. Many people don't do that either. What's the mindset that explains this behavior? You can't work backwards from the most rational thing to do to what people actually do.
Perhaps I found the article clearer because of familiarity with the subject.
On the "retroviruses must be harmless" virology: He's a denier of viral involvement in cancers in general, not just that HIV must be harmless. He is way outside mainstream consensus on all kinds of things.
For instance, he argues that Kaposi sarcoma, a very common AIDS related cancer was caused by drug use and not opportunistic infection. It is now very well established that all KS, which also affects (typically older) HIV- people, is caused by HHV-8 infection.
The core thing he does on all of these topics is just to ignore or deny anything that doesn't agree with him, eg: Hemophiliacs treated with tainted blood get AIDS, HIV viral load directly corresponds to disease progression which is clearly halted by dropping HIV load with treatment, the HPV vaccine demonstrably prevents cervical cancer, etc. He is far off in quack territory.
He also, that I know of, still supports this position. To this day, you will find people getting into this particular conspiracy and rejecting treatment. It doesn't go well for them.
I do think that freedom of speech is important, and that many attempts to squash "misinformation" are misguided, but some speech has consequences. Personally I find Duesberg utterly reprehensible and morally culpable.
The mainstream consensus is the he was wildly wrong about HIV specifically, that HIV causes AIDS, and that his influence in South Africa to not deploy anti-viral medications killed hundreds of thousands of people before the policy was reversed.
Part of his hypothesis was that viruses in general, not just retroviruses, were not connected to cancers, the consensus view is that this is completely wrong. We have a very large body of evidence on many virus caused cancers now.
So, the two known human retroviruses both cause disease and retroviruses cause diseases in animals. Duesberg held on to and promoted this concept long after it should have been clear to him that there was zero empirical support for his idea.
This injection isn't a vaccine, it's an anti-viral drug being used as pre-exposure prophylaxis. The first approval of this approach was in 2012, but using an oral pill with a short half-life taken daily.
That drug is still in use and also highly effective, the new improvement is to provide the same approach with a longer acting injected drug. One reason there has been great interest in this, despite the already effective oral PREP, is that there are thought to be socio-behavior advantages for cases like women in Africa as in this study. For example: the woman does not have to keep a supply of daily pills that a partner can find. Also possibly improved adherence with no missed doses.
The drug itself is not thought to be more biologically effective than the oral drugs, which are basically already at close to 100% effective assuming the patient actually takes them as scheduled.
South Africa (study was in SA and Uganda) has an adult HIV prevalence of 18.3% and 210k new infections per year. It is easy to select a high risk group in which you would expect to see new HIV infections during the course of the study without intervention.
No, not exactly. "the era of sending excess rooftop solar generation to the grid is over" is the OP's statement I disagree with. "They can't take any more solar input", yes they can, by adding more grid battery, which is currently happening.
If you look at that graph for a whole day you will see a huge kick-in of power company owned battery storage in the evening. There is no reason the grid cannot operate batteries at scale to absorb excess rooftop generation. NYT: https://archive.is/t1uPj
The only difference between encoder/decoder and decoder-only is masking:
In an encoder, none of the tokens are masked at any step, and are all visible in both directions to the encoder. Each output of the encoder can attend to any input of the encoder.
In the decoder, the tokens are masked causally - each N+1 token can only attend to the previous N tokens.