Show HN: Aurora – a browser engine experiment in Rust4 points·by JohannaAlmeida·3 месяца назад·5 comments
JohannaAlmeida·3 месяца назад·discussThe more updated code is here sorry . In another monorepo https://github.com/JohannaWeb/Bastion/tree/main/projects/Aur...
JohannaAlmeida·3 месяца назад·discussThank you so much . The next thing i want to tackle is the training bottleneck we have right now .That will probably be another HN post when i figure it out.
JohannaAlmeida·3 месяца назад·discussYeah RWKV is definitely related in spirit (recurrent state for long context). Here I’m combining local windowed attention with a gated recurrent path + KV cache compression, so it’s more hybrid than fully replacing attention
JohannaAlmeida·3 месяца назад·discussYeah auto complete is an amazing use case. I needed a small model that used transformers , could fit on my weak consumer GPU .So i needed to make fundamental arquitecture changes .Do some KV cache tricks.And then prove the new arquitecture was faster with benchmarks and perplexity was acceptable.
JohannaAlmeida·3 месяца назад·discussFull attention O(n²): 17.96s / 5.6 tok/sHybridAttention O(n·W + n·D): 0.35s / 286.6 tok/s