People say that about Mandarin, and really language learning as an adult in general, but the truth is most people don't put themselves in the situations to learn a foreign language nor do they exert the consistent effort required to acquire it.
The biggest problem with Mandarin is not the tones but the fact that you can't fall back on written language to help you out with anything at all. It's like learning two difficult languages at once. If you really want to learn it, though, it's entirely possible. All it requires is a human brain and consistent effort.