Ok, I've done some double checking: The Bantu expansion is approximately what I thought it was. I believe the language group I was thinking about that survived the Bantu expansion was the Khoisan.

My (very coarse) knowledge of this comes from a mixture of reading Jared Diamond (Guns, Germs and Steel) and from following it up with some Wikipedia. In short: The genetic makeup in a lot of the world is relatively dominated by the groups that were the first to adopt agriculture in their respective regions. Before the Bantu expansion, phenotypes south of Sahara were more varied, just like the phenotypes in the Americas were more varied before the corresponding "European expansion", or the equivalent expansion that happened in South-East Asia (I don't remember which society stood behind that one).

According to Diamond, we can trace a lot of (most?) surviving human phenotypes and languages back to relatively few societies, which after adopting agriculture, more or less wiped out / displaced neighbouring cultures due to increased resistance to a lot of infectious diseases and massively increased food production / need for land. This mostly happened less than 10 000 years ago, i.e. far too recently for natural selection to have a major impact on things like skin colour, hair type, height, facial features, etc. afterwards.

So: While major trends in phenotypes are of course a result of natural selection / evolutionary pressure in specific regions (resistance to skin cancer / sunburn vs. vitamin D production, or cooling down more efficiently with a wider nose vs. retaining heat with a slimmer one, or having an eye-shape that lets in more light vs. provides more shade), a lot of what we see today is simply a result of what phenotype the first group a given region that adopted agriculture had. This means that looking at the dominant phenotype in a region today will not necessarily give a good impression of what phenotype that is "optimally designed" to survive in the conditions of that region.