In practise there isn't even that solid of a connection between XX/XY and their respective associated expressions.

Cell development is highly influenced by the hormones that are present. This is especially true for sex-related cell development. This is why intersex people exist. Your genetic code just provides a bias for specific hormone production, which turns into a self-regulating loop once your cells develop into organs which produce more of their respective hormones. This is also the reason HRT is an uphill battle, and some trans people choose to have their ovaries or testicles removed.

Even biologically, the concept of biological sex makes no sense.