This was very surprising to me, so I just fact-check this statement (using Kimi K2 thinking, natch), and it's presently is off by a factor of 2 - 4. In 2024 China installed 277 GW solar, so 0.25 GW / 8 hours. First half of 2025 they installed 210 GW, so 0.39 GW / 8 hours.
Not quite at 1 GW / 8 hrs, but approaching that figure rapidly!
(I'm not sure where the coal plant comes in - really, those numbers should be derated relative to a coal plant, which can run 24/7)
Re the Apportionment Act of 1929 -- care to elaborate? Are there figures for "the worst representation in the free world"?
My impression is that there are many reasons for the dysfunction of congress; the media feedback control system (in a literal and metaphorical sense) plays an important role, as does the filibuster, lobbyists, and other corruption.
(Aside: in aging, an organisms feedback and homeostatic systems tend to degrade / become simpler with time, which leads to decreased function / cancer etc. While some degree of refactoring & dead-code cruft-removal is necessary - and hopefully is happening now, as I think most Americans desire - the explicit decline in operational structure is bad. (Not that you'd want a systems biologist to run the country.))
Seems the implicit assumption then is that M(q) -> v 'looks like' or 'is smooth like' the dot product, otherwise 'train on keys, inference on queries' wouldn't work ? (safe assumption imo with that l2 norm & in general; unsafe if q and k are from different distributions).
Correct me if I'm wrong, but typically k and v are generated via affine projections K, V of the tokens; if M is matrix-valued and there are no forget and remember gates (to somehow approx the softmax?), then M = V K^-1
I doubt it. This does not seem to be a particularly well written or well thought-out paper -- e.g. equations 6 and 7 contradict their descriptions in the sentence below; the 'theorem' is an assertion.
After reading a few times, I gather that, rather than kernelizing or linearizing attention (which has been thoroughly explored in the literature), they are using a MLP to do run-time modelling of the attention operation. If that's the case (?), (which is interesting, sure):
1 -- Why did they not say this plainly.
2 -- Why does eq. 12 show the memory MLP being indexed by the key, whereas eq. 15 shows it indexed by the query?
3 -- What's with all the extra LSTM-esque forget and remember gates? Meh. Wouldn't trust it without ablations.
I guess if a MLP can model a radiance field (NeRF) well, stands to reason it can approx attention too. The Q,K,V projection matrices will need to be learned beforehand using standard training.
While the memory & compute savings are clear, uncertain if this helps with reasoning or generalization thereof. I doubt that too.
Yes, this bothered be as well - the department of government efficiency is, as with all government agencies, is working for the public good in the public interest. This means everything must default to being open, unless there is a good reason not to be (military, CIA etc).
I don't trust Elon, and don't see why DOGE should (or could) be secret - unless it's a cover to acquire more power, which seems to be his true objective. (recently, at least)
Yep. From what I've seen, if the head wants to do nothing, it can attend to itself = no inter-token communication.
Still, differential attention is pretty interesting & the benchmarking good, seems worth a try! It's in the same vein as linear or non-softmax attention, which also can work.
Note that there is an error below Eq. 1: W^V should be shape [d_model x d_model] not [d_model, 2*d_model] as in the Q, K matrices.
Idea: why not replace the lambda parameterization between softmax operations with something more general, like a matrix or MLP? E.g: Attention is the affine combination of N softmax attention operations (say, across heads). If the transformer learns an identity matrix here, then you know the original formulation was correct for the data; if it's sparse, these guys were right; if it's something else entirely then who knows...
Not quite at 1 GW / 8 hrs, but approaching that figure rapidly!
(I'm not sure where the coal plant comes in - really, those numbers should be derated relative to a coal plant, which can run 24/7)