Thank you so much . The next thing i want to tackle is the training bottleneck we have right now .

That will probably be another HN post when i figure it out.

JohannaAlmeida

·3 miesiące temu·discuss

Yeah RWKV is definitely related in spirit (recurrent state for long context). Here I’m combining local windowed attention with a gated recurrent path + KV cache compression, so it’s more hybrid than fully replacing attention

JohannaAlmeida

·3 miesiące temu·discuss

Yeah auto complete is an amazing use case. I needed a small model that used transformers , could fit on my weak consumer GPU .

So i needed to make fundamental arquitecture changes .Do some KV cache tricks.

And then prove the new arquitecture was faster with benchmarks and perplexity was acceptable.

JohannaAlmeida

·3 miesiące temu·discuss

Full attention O(n²): 17.96s / 5.6 tok/s

HybridAttention O(n·W + n·D): 0.35s / 286.6 tok/s