The word in trans communities for this is "ew-phoria". Calling it a fetish is also kind of... Not great. Fetishes are tied into sexual gratification and gender euphoria /dysphoria is an independent mechanism that ties more closely with identity. It doesn't make trans people horny and the idea that that's what is what is happening is used often to trivialize trans experiences.

Think of the typical nature of being cis as (mostly) ascribing no value to sexually dimorphic physical features. Like you can have feelings about whether those things are attractive and the validation you receive for meeting a social standard but that's an external reinforcement system working its way inward.

Being trans is the opposite. Every sexual characteristic is not value neutral. Perceiving sex characteristics in relation to yourself either makes you feel amazing like someone has shot dopamine directly into your soul or bad like you are not actually a human. All forms of perception of these characteristics causes this effect and it is automatic and instant. Logically even if you believe these things should be valueless, obtainable by all genders if you have this feedback system you don't get to decide on how or what these things make you feel or in what proportion. The source of the feedback is entirely internal which is why it is often is at complete odds with external systems of validation and if you tried to logically explain it or lessen these feelings with logic you often just can't. The heart doesn't only want what the heart wants the heart has a shock collar on you.

In the case of facial hair particularly because it's not something you have to experience with your eyes to know it's there touch is fundamentally important. Remember dysphoria is more than being strictly about being perceived by others it is about being perceived by yourself. Other people looking at you and using your pronouns is just another way of perception of yourself. Like other people performing the job of being a mirror. Experiencing your own body however is in things like how you move, what bits of your body bump into things, your height, your weight distribution. Imagine if everytime you touch your face the slight scratch of the existence of thicker hair caused your neurological system to fire depression meds directly into your system. That's what's happening. Touch perception.

Sometimes this internal reward systems finds things about a social portion of experience that people that sucks really bad but because it equates to other people reacting to your body's sexually dimorphic characteristics when it happens to them it hits the same feel good button of other positive external recognition by a third party of having those sex characteristics.

So even while you experience the massive illogical dopamine hit from the internal reward system you can recognize logically at the same time that the phenomenon causing the reaction is a societal problem that is bad and should not exist.

Hence Ew-phoria.