Personally I don't take them to heart. I often see people ask 'Why was this downvoted?', and I get that. There are comments on here that are blatant troll-like remarks and don't add to the conversation, so it's obvious why some people get downvoted. Boggles my mind that insightful, constructive comments are downvoted though.