Thank you so much . The next thing i want to tackle is the training bottleneck we have right now .

That will probably be another HN post when i figure it out.

JohannaAlmeida

·vor 3 Monaten·discuss

Yeah RWKV is definitely related in spirit (recurrent state for long context). Here I’m combining local windowed attention with a gated recurrent path + KV cache compression, so it’s more hybrid than fully replacing attention

JohannaAlmeida

·vor 3 Monaten·discuss

Yeah auto complete is an amazing use case. I needed a small model that used transformers , could fit on my weak consumer GPU .

So i needed to make fundamental arquitecture changes .Do some KV cache tricks.

And then prove the new arquitecture was faster with benchmarks and perplexity was acceptable.

JohannaAlmeida

·vor 3 Monaten·discuss

Full attention O(n²): 17.96s / 5.6 tok/s

HybridAttention O(n·W + n·D): 0.35s / 286.6 tok/s