Definitely not a strict rule and I wouldn't want to force anyone to do it the way I do (maybe I should have marked my comment as a joke) but as far as I understand, downvotes were originally meant for spam or low-quality/low-effort comments. Stuff that just doesn't add anything to the duscussion and isn't worth reading. Fortunately, that doesn't happen very often on Lemmy.

Downvoting comments that you disagree with just to bury them, especially without even leaving a comment that explains why you disagree, just feels petty.

Overall, I'd rather upvote a well-written comment even if I disagree with its contents and downvote ten "yeah, same" comments that agree with me but add nothing to the discussion.