That's because when talking to tribalist types, you're seen as either with them or against them and in a system with 2 political parties "against them" means "supporting the other guys".

There is no independent thinking amongst the "party supporter" masses, only following and parroting of the party messages, so the idea of somebody being a genuinely independent thinker guided by personal principles rather than following some tribe or other is anathema to them.