Huge parts of South America, Egypt, and South Asia are in tropical or subtropical regions where frozen water was mostly limited to high elevations.

That said, people have always communicated so even those populations were likely to have stories about ice and snow from trading partners/travelers/cultural history from migrations even if they did not experience it directly. So millions is probably accurate, but is not as many people as it sounds when billions have lived.