I think there just needs to be a better way of communicating the reason for a downvote (or, in fact, asking for a reason for a downvote, like popup and select troll/OT/etc.) so that it costs you a little more in thought/time. It's currently just as easy to downvote as upvote, but the cost of a single downvote to the comment/user is greater than the benefit of a single upvote.
Additionally, I've been thinking for a while that it would be cool if threads were ranked a little differently, and not just on the points/time of the parent. Sometimes there are really, really great comments that are replies to low-ranked ones, and the resulting thread gets kicked to the bottom of the page. Something like Avg points/time or just accounting for the average points of next-level comments when determining rank of a reply at each level.
Additionally, I've been thinking for a while that it would be cool if threads were ranked a little differently, and not just on the points/time of the parent. Sometimes there are really, really great comments that are replies to low-ranked ones, and the resulting thread gets kicked to the bottom of the page. Something like Avg points/time or just accounting for the average points of next-level comments when determining rank of a reply at each level.