~~Just a note: The images themselves don't get federated, just the links to them (except thumbnails on posts). So the content is only on the instance it originates from. This at least makes the issue slightly better as hosting the images would be far worse.~~
Edit: Oh no, it seems the full resolution images are (at least sometimes) indeed getting cached.

Theoretically they can actually be federated when encoded as data: URI, but thankfully there's a limit of 10k characters which is basically nothing when you want to put image into it.
Example: