like what if linux still was open source but had a lot of proprietary dependencies and packages...

At that point, it's not really open source anymore. Once it has proprietary dependencies, it's no longer open.

but it still would let you use any desktop environment and there would be a new proprietary desktop environment which was like gnome but easier

What you're describing is a closed-source version of Pop!_OS with a closed source version of Cosmic, their latest DE still in Alpha.

Businesses and software companies don't make software for operating systems based on their openness or proprietary-ness. They make it based on market share. Your idea would still have to compete with Linux, MacOS, and Windows, and it would have to get a better share of the market than at least Linux before businesses would even bother making software for your closed system.

The reason Linux is as successful as it is, is because it's open, and hobbyists can and do contribute to it for free. When you close that off, you then have to pay for development, and you'll have to overcome the gigantic barrier to entry set up by the likes of Microsoft and Apple.