Maybe because you defend them. Or they see you not being a proactive force. It's not enough to stand by and wait for something horrendous to happen to try to stop. You need to be part of the culture of change which means changing how you joke and talk about things to actively pursuing to change even bad and minor habits of those you socialize with.

Anything less is condoning.