I ended up just plugging it into a speaker we already had. It isn't ideal as I also used that speaker for my record player so I have to swap the cables over when I want to play a record but other than that it works.
The Voice PE is pretty good though (other than its speaker not being great) I haven't had any issue with dropped connections and the click wheel is really nice.
Their base is starting to look pretty solid now though