Heat is mostly driven by leakage current and gate capacitance.
The big issue today is leakage currents. They typical account for around 30%-50% of total chip thermal budget, and they get increasingly difficult to control with smaller devices and lower voltages. They're also get worse with increased temperature(!).
The stacked devices here aren't the worst for leakage currents, but they're not fantastic either. Look at the 2nd graph in section 5: You'll see that the current never drops to zero over the range of gate-source voltages (for V_DS=0.7V). The minimum point is the best-case leakage current, and you can see it's well above zero! (The units on the vertical axis of the graph are unknown btw: The label reads as "current drain-source, arbitrary units")
If you have two threads on different cores that write to the same cacheline, the CPU has to enforce write ordering. The way it does this is for one of the cores to acquire a write lock on the cacheline. The actual implementation of this is via some variant of the MESI protocol that results in the cacheline on one core going to an Exclusive state, while copies of the cacheline on every other core becomes Invalid.
MESI takes a non-trivial amount of time to run. It's usually mediated at the L3 cache, which is frequently clocked slower than the main core, so it's a non-trivial number of cycles for a MESI state transition to happen (think at least ~40 cycles).
Compare that with a cacheline that's already Exclusive on a core: Writing to this cacheline is a pure L1 cache write at ~1 cycle, no MESI involved, no L3 cache involved.
Note that the typical userspace space mutex (some memory location that's modified by compare-exchange) is implicitly relying on MESI whenever two cores race for the lock, or whenever a lock is release on one core and then taken on another, etc etc.
So if your userspace coherency can be handled via some sharding that avoids needing MESI to run, it can go much faster than relying on the CPU's "internal mutex" aka MESI.
To directly address "is it possible that a hardware implementation of an algorithm could be slower than its software variant": The point is that with RSEQ, you can write a _different_ algorithm that's faster than an implementation relying on hardware mediated MESI. Instead of having memory that's accessed for a bunch of different cores, there's a chunk of memory per core, and each core almost always writes only to it's allocated chunk of memory.
Is there really that big a different in TFLOPS between the GB100 and GB202 chips?
The GB100 has fewer SMs than the GB202, so I'm confused about where the 10x performance would be coming from?
There's basically a difference in philosophy. GPU chips have a bunch of cores, each of which is semi-capable, whereas TPU chips have (effectively) one enormous core.
So GPUs have ~120 small systolic arrays, one per SM (aka, a tensorcore), plus passable off-chip bandwidth (aka 16 lines of PCI).
Where has TPUs have one honking big systolic array, plus large amounts of off-chip bandwidth.
This roughly translates to GPUs being better if you're doing a bunch of different small-ish things in parallel, but TPUs are better if you're doing lots of large matrix multiplies.
The hand-waving explanation: The slower you're going, the easier (cheaper) it is to change direction. And for eliptical orbits, the outer-most part of the orbit is where you're going slow.
So to make a drastic change in direction (aka, a very different orbit):
1. First burn to move far away from the center of the orbit so that you're going very slow.
2. Then burn to make a large change in direction (orbit).
3. Then wait until you cross your desired final orbit, and burn again to close it.
The tradeoff is that these types of orbit change are very slow (because you want to be going very slow for the middle burn, which means you take ages to get there).
Not quite: It's taking advantage of (1+a)(1+b) = 1 + a + b + ab.
And where a and b are both small-ish, ab is really small and can just be ignored.
So it turns the (1+a)(1+b) into 1+a+b. Which is definitely not the same! But it turns out, machine guessing apparently doesn't care much about the difference.
I had the same thought: Just eye-balling the graphs, the result of the subtraction looks very close to just reducing the temperature.
They're effectively doing softmax with a fixed temperature, but it's unclear that this work is going to do better than just learning a per-head temperature parameter.
The other way to think about this is that it looks like a hacked-up kinda-sorta gated attention. If that's the case, then doing softmax(alphaq_1k_1^T - log_sigmoid(betaq_2k_2^T)) might be better? (where alpha,beta are learned temperatures).
Like most things, it's more complex than that, and as a result it can be either faster or slower than 'median(RTT to each DC in quorum)'.
It's a delicate balance based on the locations that rows are being read and written. In the case where a row being repeatedly written from only one location and not being read from different location, the writes can be significantly faster than would be naively expected.
The big issue today is leakage currents. They typical account for around 30%-50% of total chip thermal budget, and they get increasingly difficult to control with smaller devices and lower voltages. They're also get worse with increased temperature(!).
The stacked devices here aren't the worst for leakage currents, but they're not fantastic either. Look at the 2nd graph in section 5: You'll see that the current never drops to zero over the range of gate-source voltages (for V_DS=0.7V). The minimum point is the best-case leakage current, and you can see it's well above zero! (The units on the vertical axis of the graph are unknown btw: The label reads as "current drain-source, arbitrary units")