This sounds like a big deal, we'll have to see if it replicates at bigger scales.
I've always felt that the position encodings being used in practice (learned position encoding or RoPE) are pretty hacky since the 'number of tokens' is not a good metric for distance between two parts of the context.
Contextual Position Encoding: Learning to Count What's Important · HackerTrans
I've always felt that the position encodings being used in practice (learned position encoding or RoPE) are pretty hacky since the 'number of tokens' is not a good metric for distance between two parts of the context.