A lot of the boundaries of today's imperium are still defined by the choices of people during the Dark Age of Technology and the Age of Strife.
The Great Crusade focused mostly on uniting worlds and systems that already had a human presence, and eliminating corruptible religions from them. For a good chunk of the galaxy, its never been colonized by humans, and for the foreseeable future, it will be left that way.
The Imperium is spread to thin, and its most ambitious leaders are focused on potential rewards for them and their worlds that they already know of. Map says "here be dragons"? Fuck it, until you find another map that says "here be treasure", "here be weapons", "here be humans making friends with xenos".
for Alan Dean Foster's "Nor Crystal Tears".