Slop.

What is the patient zero here?

Saber Marionette J was a 90s harem anime where the "Marionettes" are not human, they are robots, but they're also property.

My bet is that the trope of main character collecting a harem morphed into just a literal slave harem via "acceptable" things like "they're not human so it's ok". Combine this with other things like Pokemon where the acceptability of the Pokemon working for the humans is ok because "they love it" and they're not humans is established despite the clear ability to communicate... Then add in Isekai tropes and the trend of the isekai worlds having videogame rules... I can see the transition over time into acceptability.

Really putting the blame on any one specific thing is quite hard. But trends over time can be seen as shows influence other shows and establish new boundaries of what is acceptable or normal to put into a story.

Shield hero can't be it, because that would mean it did something remotely original, when it's otherwise such a tropey story that it seems impossible.

It kinda goes hand-in-hand with having a harem of objectified and dehumanized subordinates that Isekai is founded upon, it's just a natural development from the premise itself (unmitigated misogynist fantasizing that appeals to teenage boys in a reactionary imperialist/colonizing culture)

Isekai, at its core, the appeal is "what if I could go to this unspoiled mystical land and be superior to everyone there in some way and dominate them?" It's the colonial urge. It's not a coincidence that Isekai MCs are always godlike OP for no reason whatsoever, so that they can exert their will upon the "NPCs". And since the MC is a stand-in for the audience, what they would almost always do is create a harem of girls - the more helpless and dependent the better - that they have total power over. (Over-compensating for their past incel life where they could never get any girls)

You have the two flavors generally. The more normie-coded benevolent Isekai dictator, who merely happens to accumulate a harem of girl followers through his good deeds & the incel-coded evil Isekai dictator, who lecherously indulges in what he's doing. Some anime fans would like to say only the latter is reactionary, but really they both are. The first one is just the "rightful king" and "white/japanese savior" and "white mans burden" all combined (the Japanese see themselves as whites).

No Game No Life (which may arguably be even worse than Shield Hero, if less explicit), an isekai about a world that runs on yu-gi-oh style calvinball bullshit where the losers become slaves, predates Shield Hero by a couple of years. I haven't read this (or Shield Hero for that matter), but I did slog through the anime on netflix a decade ago; it's not good, and as bad as the anime was, well, just look up what the cover of volume 1 of the LN is. I only saw that recently, which unfortunately reminded me that the series existed at all.

Shield Hero itself may be what popularized it for isekai slop, and was probably drawing on just nebulous fantasy tropes about slavers. Or was drawing on hentai tropes, that's a very real possibility.

Mushoku Tensei is often credited as being the progenitor of a lot of the worst isekai trends, for all that it's still better than the baseline for fantasy anime in general on a lot of poitns, but it was late to the ball for this particular point. It does seem to have arrived at it on its own without copying more than general fantasy genre tropes, though.

It's wish fulfillment for middle class and downwardly mobile people who wish they were the boot on others' necks making ill-gotten surplus value rather than the other way around. That's why it has slavery, solipsistic supremacist plots, etc. in it.

I think it was Shield Hero where it took off.