HackerTrans
TopNewTrendsCommentsPastAskShowJobs

linearlayer

no profile record

comments

linearlayer
·2 年前·discuss
You seem quite upset. Can you explain exactly which quantities don't make sense?
linearlayer
·2 年前·discuss
No it's not
linearlayer
·3 年前·discuss
There is a mistake in the article, right? Multi-headed attention doesn't average together multiple attention heads. Rather it concatenates them and then right-multiplies by a matrix such that the output dimension matches the input dimension. That matrix might learn averaging, but it's not built in a-priori.