I think what you're feeling is very valid, and it's good you feel it. A veterinarian once told me that cats typically prefer to have people around them disappear than being moved somewhere else. If it's accurate, I would not stress too much about the cats at least. Also, if they travel in a pack, they have each other.
Don't forget that wherever you're going, you're gonna do good for other living beings that will like you and you don't even know it yet.