HackerTrans
TopNewTrendsCommentsPastAskShowJobs

linearlayer

no profile record

comments

linearlayer
·2 года назад·discuss
You seem quite upset. Can you explain exactly which quantities don't make sense?
linearlayer
·2 года назад·discuss
No it's not
linearlayer
·3 года назад·discuss
There is a mistake in the article, right? Multi-headed attention doesn't average together multiple attention heads. Rather it concatenates them and then right-multiplies by a matrix such that the output dimension matches the input dimension. That matrix might learn averaging, but it's not built in a-priori.