The idea is that the big bang started as a singularity, where everything had the same position as far as our 3d space is concerned, and then the difference in position arose as a consequence of it. Maybe space (as we know it) didn't exist before, maybe it did but collapsed, or maybe there was "other" space but this space we're in popped into existence. Same thing with time and perhaps some other dimensions.

So it did happen everywhere (where everywhere is just everywhere inside this universe), but it was a single point at that moment.

Though I suspect it was inside another universe and that our big bang singularity was just another black hole forming in that universe and we're seeing the mystery of what happens beyond the event horizon when gravity overpowers all other forces. Our familiar forces could just be the next set of rules physics for small things (from the perspective of the parent universe) after gravity overcomes the dominant ones in that universe. Which could mean that all black holes are tunnels to other universes (that we can't visit but the matter that makes us could, though it would probably be something else once it did, like an entire galaxy cluster).

Then the CMB might just be light that entered our universe from the parent one, redshifted like crazy (plus other optical gravitational effects, like any light that enters will appear brighter in one direction but coming from all directions to some extent).