The RWVK modal was mention which is not based on transformers but on NN. [1]
The context window is particularly interesting, i have interacted with the people over discord some time ago and the model seems good but not widely used yet.
People are noticing the limitations will not shift to pure hardware -> energy now.
the transformers allows heavy parallelization but it's too computationally-intensive even with quantitization.
people are simply trying to run from the transformer is seem.
i sort of specialize in meta cognition(zettelkasten, spaced repetition, incremental reading, speed reading etc)
i believe the effects of writing by hand are so maginal you would re-gain it by not having to re-type your notes. putting them into spaced repetition etc.
just make everything digital and then OCR the rest.
why don't we build a 'line' in the pacific + indian ocean and carry containers by a small floater between countries in a 'corridor', yes they would be 'explored' to salty water but i think this is not as bad ...
we could then do 'gaps' between the containers as to let other ships cross the oceans
we could increase our capacity by 10x? and adding more is just adding another 'lane'
we would only have to worry about pirates but i think we can handle that
most people don't care/want to finetune. they want their model in a cute and easy interface.