I do not think so. Your directional hearing depends on the shape of your face too. This means that everybody hears directions differently, because their face is different. I think people should be able to adapt to these smaller changes over time.

But I'm not sure what I'm saying is true. Games usually have binaural sound, but it may not be effective for some people, because the head-related transfer function (HRTF) that the game uses sounds different compared to what somebody would hear irl.