It's certainly an odd concept the more you think about it, especially because of how relatively recent it is (only a few hundred years). Before that, discrimination was based on culture and education. Ancient Greeks called outsiders barbaroi not because of their appearance but because their languages didn't sound as sophisticated as Greek. Romans had multiple African emperors and it was a non-issue because they were culturally sophisticated and spoke good Latin.