Right now, the Earth is losing mass at about 55 000 tons per year. Yes, losing. About 100 000 tons of hydrogen and helium escapes the upper atmosphere, partially offset by roughly 45 000 tons of dust and meteorites getting scooped up along our orbit.

Considering this has been happening for millions of years, I think we're quite safe from affecting the Earth's mass and orbit within the span of even centuries.

But it's much more likely that the majority of material mined and processed in space will not be coming down to Earth. It's much better put to use in orbital construction, or shallower gravity wells like the Moon and Mars.

You're entirely right that getting to the rocks, and getting the mined stuff to where it's actually useful, are gonna be a problem. Maybe we'll finally get some nuclear thermal engines, cause the shite ISP of chemical rockets is really insufficient for these trips and ain't no one wanna wait on the gravity assists.