It's a problem because there are perverts in sheep's clothing that hide under identity/gender and will go to other bathrooms as an excuse to perv on the opposite gender.
Would you feel comfortable if you were a woman and some creep of a guy with feminine-like features but still ultimately a guy, came in to the women's bathroom just to try and sneak peek for nudes?
I mean, if someone does that, it says a lot about someone. So not only are they a perv, but are they also into the idea of someone shitting and pissing? Technically I say you are, if you do this.