HackerTrans
TopNewTrendsCommentsPastAskShowJobs

linearlayer

no profile record

comments

linearlayer
·há 2 anos·discuss
You seem quite upset. Can you explain exactly which quantities don't make sense?
linearlayer
·há 2 anos·discuss
No it's not
linearlayer
·há 3 anos·discuss
There is a mistake in the article, right? Multi-headed attention doesn't average together multiple attention heads. Rather it concatenates them and then right-multiplies by a matrix such that the output dimension matches the input dimension. That matrix might learn averaging, but it's not built in a-priori.