it's a bit slippery slopey but yes i suppose so. if your downvoting sprees amount to being abusive as the admins or mods define it then on their terms your account should be banned, ip banned, whatever. i don't think there's any voting based behaviour that gets you in irl trouble, obviously posts inciting violence, sending credible threats or other behaviour that gets you noticed by your regional authorities is another kettle of fish, but none of that seems relevant to up/downvotes being visible