You didn't forgot the butler doesn't have the key. When you hired the butler you already gave permission to let the butler cast that magic linking from the dollhouse to the safe already (so that you don't have to keep giving the butler the key when he needs to create new dollhouses that can move stuffs around the safe).

The issue here is that this butler don't have another separate security protocol to make sure you are the one who can limit what he does with the dollhouse. Any guest that are already invited to your house can ask the butler to do things, so they get to touch the contents of the safe even without the key to the safe.