You're right, it's the intent that matters. <any_word>, used to describe something stupid or negative while also being an outdated description for a specific group of people...
The fact is, it's _that_ word that's evolved into something hurtful. So rather than be the guy who sticks up for the_word and try convince everyone it shouldn't be hurtful, I just decided to stop using it. The reason why I stopped was seeing first hand how it affected someone with Down Syndrome who heard me saying it. Sometimes real life beats theoretical debate. It's something I still feel shame about nearly 20 years later.
It wasn't a particularly onerous decision to stop using it, or one that opened the floodgate of other words to be 'banned'. And if someone uses it and hasn't realized that, then move on - just avoid using it next time. Not a big deal. It's the obnoxious, purposefully hurtful use of it that's not great (which doesn't seem to be the case here tbh). It's the intent that matters more.
Might seem like a descriptive word but the fact is, it's hurtful to people who are working harder to make their way in life than I'll ever have to. Even when just heard in passing.
Why do things in life that will hurt someone who'll likely just retreat away rather than confront you. Be the good guy.