Others summed these points up quite nicely, but:
-
The sand and remoteness makes installation and maintenance difficult.
-
Heat is hard on electronics.
-
At a huge enough scale, it would lower the albedo of the (normally very reflective) desert, heating up the local environment.
-
Transmission. This is the biggest factor. Transmitting tons of power a long way is tremendously difficult, dangerous, slow to build, expensive, and unavoidably inefficient.
-
On top of that, you'd want to buffer power (with batteries) or keep spare capacity (like gas generators) near the source to regulate the supply. This is less of an issue at smaller scales, where solar 'blends' into the grid.
-
Producing the things. It's easy to say 'just make more', but you'd need to massively scale up every section of production: the mining, the transportation of the ore, the transportation of the panels, the production of the machines to fabricate the things, educating people to make those machines, all in a frayed global supply chain.
-
Countries might feel uncomfortable being so dependent on each other for energy. Yes, the irony is tremendous.
It's not impossible. There are installations in the Atacama Desert (for instance), which is basically the best case scenario (ridiculously dry, high altitude), but you'd run into problems scaling it up to, say, power all of Brazil.
There are reasons power generation tends to be more localized. Adapting it to the local environment, a la carte, is kinda the way to go.