When I was a younger man I used to feel very uncomfortable around gay people, particularly gay men. I didn't "hate" them per se, but being around them made my skin crawl. This was a direct result of the environment I grew up in, and the lack of experience I had with actually getting to know gay people personally. Eventually once I made my way out into the world I was forced to interact with actual gay men and learned that they are just normal people, and my discomfort was entirely in my head, it was a product of ignorance and prejudice. I am glad to say I am better for having moved beyond that. At that time those feelings were a lot more politically valid, but that doesn't make them any less homophobic. What you are validating is the same. Trans women using female bathrooms pose no actual risk to those cis women (any more than other cis women do in female restrooms). I would bet that very few of those cis women vocalizing those concerns have ever actually been aware of sharing a restroom with a trans woman. They just imagine some hairy boogyman dressed in a dress saying "I have the right to use this bathroom" but that's in their head. It's not a valid fear, it's a phobia. They need to be educated, not coddled and told those feelings are valid. If trans women were actually assaulting or peeping on cis women using the restroom that might be different, but the fact remains women (cis or trans) are almost certainly much more likely to be assaulted in the women's restroom by a cis man than by a trans woman. You can't say you are not trans-phobic and still validate those baseless fears.