The different servers, having to remember other people's instances along with their username.

This is just like email, I see no problem here.

I think the problem is about the mindset and the onboarding experience. We've used too much proprietary products and prefer something easy and not too much diverge from the norms. Recently, I tried to advertise Mastodon and Lemmy to my non-techie friends, which are using X and Reddit. Some did try but gave up. They said they didn't understand the concept, and didn't want to bother with choosing an instance in the first place, because they didn't understand the federation concept. It's just hard to explain the benefits of the fediverse to non-techie people.

The type of people that the fediverse attracts are FOSS users.

I have the same observation as your view. Current fediverse communities are heavily towards tech. Some of my friends joined but gradually left because they had a few to no interactions or no interesting people in their interested areas to follow.