Fair point, the gap isn’t huge in that plot, and both degrade at low ratios.
The difference is more in how they degrade: TopK can have sharper, localized failures, while HAE tends to be a bit more smooth. That doesn’t always show up strongly in average MSE.
That said, the gains are modest right now, this is still a research prototype exploring the tradeoff, and there’s clearly more work to be done.
Yeah, the latency hit is definitely real. That said, most of what I’ve run so far is CPU-bound, which likely exaggerates it quite a bit so I didn’t want to draw strong conclusions from that.
Would need proper GPU implementations to really understand where it lands.
I completely agree.Right now this is all on a synthetic setup to isolate the behavior and understand the reconstruction vs memory tradeoff. Real models will definitely behave differently.
I’ve started trying this out with actual models, but currently running things CPU-bound, so it’s pretty slow. Would ideally want to try this properly on GPU, but that gets expensive quickly
So yeah, still very much a research prototype — but validating this on real models/data is definitely the next step.
That’s a great point and yeah, I’d agree SVD itself isn’t new at all.
On downsides: definitely a few. The biggest one is latency - SVD is fairly heavy, so even though it’s amortized (runs periodically, not per token), it still adds noticeable overhead. It’s also more complex than simple pruning, and I haven’t validated how well this holds on real downstream tasks yet.
This is very much a research prototype right now more about exploring a different tradeoff space than something ready for production.