Towards understanding multiple attention sinks in LLMs(github.com)
github.com
Towards understanding multiple attention sinks in LLMs
https://github.com/JeffreyWong20/Secondary-Attention-Sinks
https://github.com/JeffreyWong20/Secondary-Attention-Sinks
The converted sinks are termed secondary attention sinks as they are weaker then BOS attention sinks.
This might be related to layer specialisation in LLM!