If you don't want your comments hidden - don't say shitty things. It isn't a hard concept. This also applies to being banned.
I'd like to think my comments are reasonable 99% of the time (or at least I aim to post reasonable comments), but my comments get downvoted much, much more than 1% of the time. Most of the time, I'm either completely baffled as to why my comment got downvoted or I know it's because some people don't like my opinion (which I usually try to present in a reasonable and in a relatively well-argumented manner). At the very least I know I don't 'say shitty things'. Personally I find downvotes on comments both annoying and very discouraging.
In case it wasn't clear, I strongly oppose having downvotes on comments, and I think they should be removed altogether. I like them in theory but practice has shown they simply don't work.