Benchmarking deep learning activation functions on MNIST [OC](heartbeat.fritz.ai)
heartbeat.fritz.ai
Benchmarking deep learning activation functions on MNIST [OC]
https://heartbeat.fritz.ai/benchmarking-deep-learning-activation-functions-on-mnist-3d174e729735
6 comments
A more in-depth paper about this found the Swish activation often outperformed other functions: https://arxiv.org/abs/1710.05941
Most of the recent research is moving to GELU (Gaussian Error Linear Units) activation functions: https://arxiv.org/pdf/1606.08415.pdf
That's interesting. I didn't read the paper closely, but skipping to the pictures, it looks like ReLU, but smoothed out so the derivative is continuous. Intuitively, that seems useful.
I wasn’t aware of that one. Definitely interesting, thanks for sharing!
These are small differences, and this is on MNIST (very small toy dataset). This is likely just noise. How big is the variance when each experiment is tried with different random seeds? And more interestingly, how about more difficult problems? E.g. try out on some real world tasks, like e.g. speech recognition (e.g. Librispeech). I don't think you can draw any conclusion from these current results.
Thanks for the advice!