Thanks for the tip. I'll see where I can apply it.
Most of the time is usually spent in the convolution layers. Convolution is not a matrix multiplication in the current implementation. I guess it would be a matrix multiplication in frequency domain or by using a Toeplitz matrix.
I've implemented a CPU Parallel version and gave a try at GPU implementation. But I'm not satisfied at all by the GPU version :)
While visiting a lab in Paris, I saw some researchers using neuroevolution to teach a bird-like robot how to fly.
some links related to neuroevolution:
https://en.wikipedia.org/wiki/Neuroevolution_of_augmenting_t...
http://eplex.cs.ucf.edu/hyperNEATpage
https://github.com/colgreen/sharpneat