All you have to do is look at Reddit, which is worthless for actual discussions. People with dissenting opinions often don't bother leaving a comment. It creates an environment of exclusion by providing a negative stimulus for anyone not in the majority. Worse--as we've already seen in this thread--it establishes the thought process of "if it was downvoted then it must have been wrong," which is kind of frightening. No, I don't expect this site to brainwash people the way Reddit does, but it's still icky. The online experience has become all about reinforcing the hivemind mentality and driving away or censoring anyone who diverges from the herd.
At the very least, there is no need to hide the button that reveals downvoted comments.