Host to device bandwidth (ram to vram) is 128Gb/s for PCIe Gen 6. VRAM to GPU bandwidth is 1.8Tb/s for GDDR 7 (5090), and 8Tb/s for HBM3e (B200). So it can be faster to recompute than offload kv cache.
The Little Learner is a great book. I tried rewriting all the code in Python/JAX while following the Scheme code style as closely as possible, and it worked out great.
The appendix on autodiff is a bit rushed, in my opinion. But in all fairness, the number of pages would probably need to be doubled to give a proper exposition of autodiff.