Simple, zero overhead way to compress model, KV cache via Low-Rank Decompositionjeffreywong20.github.io1 points·by thw20·2 เดือนที่ผ่านมา·0 comments
Towards understanding multiple attention sinks in LLMsgithub.com1 points·by thw20·4 เดือนที่ผ่านมา·2 comments
The Existence and Behavior of Secondary Attention Sinksarxiv.org1 points·by thw20·5 เดือนที่ผ่านมา·0 comments
thw20·3 เดือนที่ผ่านมา·discussGood work! This is very interesting. Here's a related work that construct low-rank approximation for attention: https://arxiv.org/abs/2505.12942.Maybe the idea of Query calibration matrix Rxx is of interest to the author!
thw20·4 เดือนที่ผ่านมา·discussThe up to date paper documenting and analysing the observation is now available on arxiv!
thw20·4 เดือนที่ผ่านมา·discussThis project reveals an interesting phenomena, where LLM converts semantic non-informative tokens to attention sinks through middle layer MLP.The converted sinks are termed secondary attention sinks as they are weaker then BOS attention sinks.This might be related to layer specialisation in LLM!
thw20·4 เดือนที่ผ่านมา·discussThis is so amazing. What a masterpiece for intro to reinforcement learning in llm.
Maybe the idea of Query calibration matrix Rxx is of interest to the author!