While part of me agrees with your analysis, I'd like to point out what I think could make this wave of ML/AI more serious. You are absolutely correct that deep learning is not very biologically accurate and that what today's models do seems a long way from AGI. However, in my opinion, the most fundamental aspect of intelligence is the ability to form useful abstract ideas to model reality. To make that more concrete, as a rather extreme example, consider the invention of numbers. The process by which people developed the notion of abstract quantity separated from any particular real experience is, to me, the most archetypal example of what it means to be intelligent. Of course, deep learning can't invent abstract math, but it seems to be able to mimic this process in a very rudimentary way. It's not a faithful representation of real neural networks, but perhaps it has just enough of the right ingredients, scale, depth, non-linearity, hierarchy, such that it is able to demonstrate a spark of that magic, hard-to-define process of intelligence. When a deep net learns MNIST, it seems to come up with an abstract notion of what a handwritten 9 looks like and it's hard to argue that there isn't something very mysterious and special happening.
The importance of math for machine learning didn't fully sink-in for me until I developed a deeper understanding of the back-propogation algorithm that makes training deep neural networks computationally feasible. If it weren't for the introduction of a continuous activation function combined with the nice properties of the multivariate chain rule, we probably wouldn't be having this discussion at all. Another example that comes to mind is kernel methods used for Support Vector Machines.
No matter where the inspiration for a model comes from, the final formulation is always mathematical, and I think that without an appreciation for the mathematics, it's hard to get a true feeling of a model's effectiveness and limitations.