I think the reason why it hits so hard is because it's an attack that lands under their own ideology. To them, being outside the normal is the worst thing you can be. They demonize everyone that they perceive as outside the normal. So, when they're accused of not being normal, they take that very seriously.

Most other attacks against them don't bother them as much since they're based in other ideologies and don't really land under their own ideology.