This age period is shockingly young; I don't remember exactly but iirc it's less than one year old.

Anecdotally, that sounds hard to believe to me, because I moved from South Korea (which has the same sound ambiguity) to the US at the age of 11 and have no problem at all distinguishing between the two sounds. All of my Korean friends that I had back in middle school also had no issues, even though none of them were born in the US. On the contrary, I've been told at times that I speak Korean with an obvious western accent.