but by this logic wild animals should all be kept in zoos

Domestic housecat is not a wild animal any longer. It's closely related to wild species, but it is different. It's a domesticated animal, meaning humans changed it over generations. This is the same for dogs, cows, pigs, chickens and most other animals we keep for food or work.

We don't and we shouldn't let domesticated animals loose in the world intentionally, barring a few exceptions for controlled grazing. They no longer have a place in the ecosystem and releasing them in sufficient numbers could greatly upset it as we've seen with domestic cats.

Especially since domestic cats don't need to hunt. People who let them out provide them shelter and food. Therefore the only reason cats get out is to prey upon local wildlife for entertainment. That's akin to letting your dogs out to run around in packs murdering any squirrels, rabbits or cats they find. Or letting your sheep or cows wander from your farm into the local town where they cause problems with traffic.

There is no reason to let your domesticated cats out to wander except for a misplaced sense that they "should".