Hi, I have read your interesting paper. I recommend you our previous HyperZZW paper (https://arxiv.org/pdf/2401.17948). I think there are a lot of similar concepts here.
1. Context-dependent convolution
2. Global & Local branches
3. Replace large-filter Conv with matrix multiplication
4. Information bottleneck -> Information loss
I also want to share that Mamba is based on the concept of Hyena. And the simplicity is the best (HyperZZW), and Hyena is a failure.
1. Context-dependent convolution
2. Global & Local branches
3. Replace large-filter Conv with matrix multiplication
4. Information bottleneck -> Information loss
I also want to share that Mamba is based on the concept of Hyena. And the simplicity is the best (HyperZZW), and Hyena is a failure.