Long-term structure
MuseNet uses the recompute and optimized kernels of Sparse Transformer to train a 72-layer network with 24 attention heads—with full attention over a context of 4096 tokens. This long context may be one reason why it is able to remember long term structure in a piece
> We have no atmosphere of innovation because our salaries are garbage compared to the US. This is the problem, every single time. Most of my Canadian graduate friends work for US companies, some remotely. We literally do not have economic reasons to stay here. No amount of lip service will ever transform 100k into 300k.
> Mr. Meadows said he was later dismissed for what he was told were “performance issues.” Tesla declined to comment on Mr. Meadows but noted that the incident happened months before the release of the technology, giving the company plenty of time to work out problems that had been discovered during test drives.
We used to treat our test pilots with the highest regard. How low have we fallen?
Long-term structure MuseNet uses the recompute and optimized kernels of Sparse Transformer to train a 72-layer network with 24 attention heads—with full attention over a context of 4096 tokens. This long context may be one reason why it is able to remember long term structure in a piece