It isn't that there's tons of trans athletes... It's that even at fairly low levels of sport there are currently more options available to people with disabilities to participate then there are of people of intersex and trans backgrounds. In a lot of cases tracking performances of trans athletes they aren't dominating. There's stories of transfem athletes who regularly sit around getting 15th place but after coming in first one time the entire sporting becomes hostile to trans people.

In civil rights discussions there's a concept of rights of participation. The concept being that being barred from social, political or recreational spheres creates outsized harms on the ability to make the advantageous connections others are given free access to and creates classes of segregation.

There's also a catch 22 situation. If someone opts to go through a trans puberty instead of a natal one there is no meaningful difference to speak of between the physicality of trans athletes and cis ones. If forced to stay inside their original sex segregated sport not only are trans people being being told in no uncertain terms that society does not accept their new status regardless of parity, they essentially become isolated inside the sporting body. Either you have someone whose body is feminine placed in a sport with only cis males to be compared to or you have a masculine body placed inside a group with all cis women and both will be framed out of being taken at all seriously inside the entire body of that sport. A lot of trans people can't participate in sport not because they aim to be picked for any of the social leg ups excellence in sport provides... But for any of the regular benefits of just participating.

It creates a fair sting to have a government force your choice of initial puberty that neither you or your doctors and parents thought was a good idea... and then sit back and watch the rest of society constantly punish and isolate you for going through that puberty by then treating you as a logistical social problem for the rest of your life.