The problem with simple rules is it removes all nuance. For example, when smoked tobacco is extremely addictive and therefore harmful according to you. But then you would ignore all the other practical, medicinal uses that tobacco has when used with wound dressing and pain relief.
Of course that's not a perfect comparison to coffee and caffeine, but just because you've got people hard-addicted to Starbucks lattes and espresso doesn't mean caffeine is flat-out harmful to us all.