Maybe a bit outdated now, but reminds me of LSTMs from the recurrent update of a memory / hidden state with gating. I remember one of the biggest problems with such RNNs being vanishing gradients as a result of the long context, which vanilla transformers presumably avoided by parallellizing over the context instead of processing them individually. I wonder how this is avoided here?
Love the site, but the one thing I am missing is instantly seeing what papers are trending right now, instead of having to manually select a timeframe. Think hacker news frontpage. Would be interesting to add a "hot" filter, or similar. Average weighting by exponential decay over time? Not sure how you usually do that with pagerank.
http://nobelprizes.com/nobel/why_no_math.html#story