I think they tried it already in the original transformer paper. THe results were not worth implementing.
From the paper(where Additive attention is the other "similarity function"):
Additive attention computes the compatibility function using a feed-forward network with
a single hidden layer. While the two are similar in theoretical complexity, dot-product attention is
much faster and more space-efficient in practice, since it can be implemented using highly optimized
matrix multiplication code.
Can someone elaborate on how this translates to the actual performance of GPU. What is the performance jump (will this allow to run 4k x 4k per eye VR headset) when compared to current GPU generation ?
Today Even with Nvidia 1080 cards running 2k x 2k per eye headsets is not an easy task and beyond a smooth VR experience.
I tasted kofola as a child and I did not liked it all. To me it tasted like a weird mixture of tea and coca-cola and I found it disgusting. Coca-cola tasted much better. Later as an adult i tried kofola couple of times and again did not like it. As the time went by somehow I get used to it, and now i go for kofola over coca-cola everytime. I dont expect anyone trying it for the first time to like it.
From the paper(where Additive attention is the other "similarity function"):
Additive attention computes the compatibility function using a feed-forward network with a single hidden layer. While the two are similar in theoretical complexity, dot-product attention is much faster and more space-efficient in practice, since it can be implemented using highly optimized matrix multiplication code.