Voting provides feedback and a way to democratically determine something's visibility. I would call it a stretch to say something is invalidated just because it's unpopular. The only benefit to having a popular post/comment is better visibility. The points themselves have no value beyond what you personally give them.
Some people will filter what they say to exploit this system, but no system of even modest complexity would be fully immune to that. If there is a problem, it's the users having poor judgement, but it's beyond the scope of a forum sorting system to save people from themselves.
Same for me. A few comments are unlikely to be persuasive by themselves but they are great at introducing me to new perspectives. It's not uncommon with some follow up research that I do change my mind, and comments are what get the ball rolling. Especially true for topics I haven't thought much about.