There is a potential limit beyond the budget - communications.

I found this article talking about how they're making jam-resistant drones: https://www.popularmechanics.com/military/a42922481/tricopter-drone-atlaspro-resists-russian-jamming/

So the original method of jam-resistant secure comms was spread-spectrum technology, which consumer drones use these days. Apparently modern jam-resistance involves frequency-hopping to unjammed frequencies, but it's an arms race. I assume it involves things like the jammers monitoring for frequencies that are being used and switching to jam them, which basically means they're playing whack-a-mole with the frequencies. Jamming every frequency is very energy-intensive and also blacks out your own comms, which could be exploited to make you go dark by simulating a drone attack and instead of actually attacking, using the enemy's own jammer as cover. There's always something to exploit.

Anyway, they apparently already are doing swarm communications with up to 5 drones, and they're trying to up that to about 50 or so. I guess part of the challenge would be that when you've got that many signals flying around the airwaves, it's a lot easier to jam at least a portion of them. That makes your jamming issue a lot harder. I mean clearly they don't think it's impossible because they're trying it, but you might not be able to do hundreds or thousands for instance.