I beat him by cheese in a different way. If you follow the panther, it will lead you to the room where the fight starts. But, if you skip that room and go further down you'll end up in another area with a structure that looks like a gazebo with a large spider corpse inside.

You can go all the way to the back of this room where there is a chest and wait for the panther to nibble on the spider. Kill it and then above you on the left is the boss with his little army. You can trigger the agro from below by shooting arrows, world, or throwing things up and getting the mobs to approach the ledge above you. Just stay back and target whatever you can on the edge and they'll keep standing there until they're all dead.

You can sometimes get the boss to stand there too or you can get him to come down and at fight him solo which is quite manageable.