Your own answer offers a far better example of disingenuity, at least so I feel.

It is socially acceptable not to date someone due to a biological trait (of which being trans is a prime example) you are not attracted to (i.e. personal preference), however (I certainly believe) it isn't really socially acceptable to say "I don't want to spend time with X because of " (your action is motivated by a personal preference). One is a clear matter where personal preference trumps, but the other is one where polite society forms an interesting grey area - where between those two is your line?

I'd disagree with the statement that you can be a nazi in your own home (a good strawman there), since that just means you'll be training a nice younger crop of nazis (which is the real root of the issue), but that isn't the question at stake here. It's "why isn't everyone up in arms against transphobia", and the answer is that no-one can agree on where the line should be drawn, and most people are worries that it'll turn out like every other attempt to stamp out particular ideals. However, in the spirit of charity, my line is drawn well before the person begins to shout "gas the Jews" in their own home, because prior to that the harm was already done.

Hate speech definitely removes freedoms and rights, but it is the ears that listen that determine whose rights and freedoms are removed. Will the crowd turn upon the person spewing hate (giving in to a morally acceptable hatered, thence rises the paradox), or will it follow the voice guiding them towards hatered.