functionally the downvote button is the “I had a negative reaction to this button” and the up vote is the “I had a positive reaction to this button”

Exactly this.

Some of my favorite comments have loads of downvotes and to me it just makes whatever dumb shit I said funnier

If you're talking about your comments, and you laugh at others who downvote them, then well done. But I submit that most humans are not so thick-skinned.

There's also the question of visibility: downvoting a comment is not just saying "You're wrong", it's making the comment less visible to others. A form of censorship, basically. I personally find it extremely annoying and unfair when it happens.

Perhaps it also has to do with how much time and thought you put into the offending comment.